Running g2_tracking (BMAD) on the fermilab computer grid

Go to your main directory in the path /gm2/app/users/< username >/
Create a working directory. Let's call it work
Go to /gm2/app/users/< username>/work
Copy the following to this directory
- input.dat
- lattice file, (i.e. bmad_grid_quads.)

Creating the tarball

Create a subdirectory /gm2/app/users/<username >/work/bmad_grid

go to /gm2/app/users/<username >/work/bmad_grid. We are going to put all of the files that g2_tracking needs to run in /bmad_grid. This will include executable, a special library file, as well as all of the magnetic and electric field files, lattice file and input file. To simplify the task of recreating everything you need in this directory

Run /gm2/app/users/drubin/bmad_dist_fix/g-2/softlink_gm2 . This will create softlinks to almost all of the files you will need

ln -s /gm2/app/users/drubin/bmad_dist_fix/production/bin/g2_tracking
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/libncurses.so.6
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/pulse_127cm.dat
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/magneticfield/ring_inject_bfield3d_wuzheng_extended.dat 
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/magneticfield/inf_field_alone.dat 
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/muons/VDstop_DS_436_12000.dat 
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/muons/particles_endm4m5_100.txt
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/KICKER_E989_NoRolledEdges.dat 
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/quads/
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/calorimeter_lattice.
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/fiber_monitor_lattice.
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/walls_free.bmad
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/walls_quads2-4.bmad
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/backleg/
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/bmad_all_configs_fringe_inf_grid.
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/bmad_grid_fringe_inf.bmad
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/magneticfield/BrFourier2016.dat
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/magneticfield/BzFourier20170628_LogID983.dat
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/g-2_FFT_04032015.lat
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/g-2_FODO_04032015.lat
ln -s /gm2/app/users/drubin/bmad_dist_fix/g-2/files/parameters_g-2_03112015.lat

Then copy ../input.dat and ../bmad_grid_quads. to bmad_grid/.

Checking that the tarball is complete

This would be a good time to check and see if you have everything you need. In /work/bmad_grid type

 source /gm2/app/users/drubin/bmad_dist_test/mybashrc.   (to set the proper library)
 g2_tracking  (or maybe) ./g2_tracking

Create a tarball with the command (you are still in /work/bmad_grid/)
```
 tar --dereference  --exclude *.svn * -cvzf ../bmad_grid.tgz
```
Go up ../work/. You will find bmad_grid.tgz in this, (the /work/) directory.
Stay in the /work/ directory and copy the following files into it
/gm2/app/users/drubin/bmad_dist_test/g-2/files/submit_bmad_grid.sh
/gm2/app/users/drubin/bmad_dist_test/g-2/files/setup
/gm2/app/users/drubin/bmad_dist_test/g-2/files/run_bmad_grid.sh

Submitting the job

You are almost ready to submit

/work

Type source ./setup
Type submit_bmad_grid.sh (Change this file to specify the # of jobs you want)

Output

But before submitting the job, you should set up the output directory

/pnfs/GM2/scratch/users/ <username> /bmad/

The output will be written to a subdirectory named by date and time. You will be advised of the name when you submit. The stuff that is the standard output ends up in subdirectory /logs/ All of the g2_tracking output files are written into a new directory with name with date and time that the job started. The directory and its contents are in a .tgz file in subdirectory data
when the job is completed, copy the tar file and the log file to /work/ and then type
- tar -xzf output_<number>.tgz
This will create a subdirectory of work/ with all of the output files.

Keeping track of the job

Check the jobs with jobsub_q --user=<username>
or if you want regular upates watch -n 2 jobsub_q --user=<username> -- will update every 2 seconds
- State = I means waiting to run
- State = R means running
- State = C means ending
If you need to kill the jobs, do jobsub_rm --user=<username>
If problem, look in the *fifebatch* directory for the *.err file
If the job dies and you want to know why, go to the scratch area and do... sh fetchlog.sh -- -and then go to the directory that it said it made. Then look at the *.err file.