Page 1 of 1
Tried to implement simple barrier in linux shell but fail
Posted: Mon Sep 21, 2020 7:38 pm
by ggodw000
I think I am doing something wrong when I tried to implement simple barrier in linux shell script but seem to fail spectacularly.
Code: Select all
# some code block before launching scripts in parallel.
<snippet1>
...
for loop
# should launch instances of each script1 in background since ampersand implied.
launch another shell script in <script1> in background &
done
#barrier implementation, wait loop until all instances of script1 finished running.
for loop
count=`ps -ef | grep <script1> | wc -l`
if count -eq 0
break
fi
done
...
# some code block after launching scripts in parallel.
<snippet2>
The problem is I can see <snippet1> and all instances of <script1> is launched
But output of code block in <snippet2> appearing before the output of instances of <script1>.
Because there is a dependency where <script1> can not run after <snippet2> is executed it runs into all sort of errors.
Re: Tried to implement simple barrier in linux shell but fai
Posted: Mon Sep 21, 2020 8:26 pm
by Octocontrabass
Try replacing your barrier with this:
Re: Tried to implement simple barrier in linux shell but fai
Posted: Tue Sep 22, 2020 5:50 am
by bzt
ggodw000 wrote:Code: Select all
count=`ps -ef | grep <script1> | wc -l`
This always going to be larger than 0, because grep itself is included in ps output. (btw the flag "f" makes no sense here.)
Code: Select all
$ ps -ef | grep something
bzt 780842 147715 0 13:44 pts/0 00:00:00 grep something
This trick I've learned from an old UNIX guru:
This will list all processes containing "something" except the grep
The reason is, the command will be "grep
omething", however [] is replaced by the shell, and "grep something" will actually be executed, that doesn't match "omething".
Btw, "wait" is a much better solution, I agree.
Cheers,
bzt
Re: Tried to implement simple barrier in linux shell but fai
Posted: Tue Sep 22, 2020 7:58 am
by nullplan
bzt wrote:This always going to be larger than 0, because grep itself is included in ps output. (btw the flag "f" makes no sense here.)
Not necessarily (and that can lead to some fun bugs): The shell will spawn all parts of the pipeline in parallel. So whether "grep" has been exec'd by the time ps gets to that process is not defined. There is no sequence that makes that a necessity, and indeed, in some situations ps fails to output the line with "grep" in it (if grep happens to get a very small PID and ps has already iterated over it by the time grep is exec'd). And so you end up with the grep line sometimes being in the output and sometimes not. Happy debugging!
The trick you showed is a good one, however. I already knew it, but it is a good one for newbies (regular expressions don't have to match themselves). Alternatively, you can filter the "grep" line afterwards with "grep -v grep". But for this particular problem, Octocontrabass was entirely correct: There is no need to iterate over all processes when you only need to wait for your child processes.
Re: Tried to implement simple barrier in linux shell but fai
Posted: Tue Sep 22, 2020 8:55 am
by ggodw000
thanks I used wait and apparently working much better.