Possible bug in the "Average Processing Rate" calculation.

Message boards : Number crunching : Possible bug in the "Average Processing Rate" calculation.

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95869 - Posted: 2 May 2020, 21:38:23 UTC - in response to Message 95864.  

The estimated run times that the client provides can be off and irregular. But the job cache should hopefully now be more accurate given the deadlines and cache settings. Let us know if that is not the case with your current settings? It looks ok to me since .2 + .2 days of work = 9.6 hours and 10 work units = 10 hours with a cpu run time preference set at 1hour (the default).

The feeder part of the scheduler was updated to hopefully send a more reasonable number of jobs depending on the deadlines, cache settings, and existing cache, based on the cpu run time preference. This change will not affect the estimated run time information that the client provides which can be off. This estimate that the client displays is just a rough estimate to provide the user some feedback but is not used for anything else to my knowledge. So you can ignore the run time estimate/remaining time estimate/percent progress and assume the job will run close to the cpu run time preference.
ID: 95869 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95870 - Posted: 2 May 2020, 21:47:41 UTC - in response to Message 95869.  
Last modified: 2 May 2020, 21:52:35 UTC

The estimated run times that the client provides can be off and irregular. But the job cache should hopefully now be more accurate given the deadlines and cache settings. Let us know if that is not the case with your current settings? It looks ok to me since .2 + .2 days of work = 9.6 hours and 10 work units = 10 hours with a cpu run time preference set at 1hour (the default).

The feeder part of the scheduler was updated to hopefully send a more reasonable number of jobs depending on the deadlines, cache settings, and existing cache, based on the cpu run time preference. This change will not affect the estimated run time information that the client provides which can be off. This estimate that the client displays is just a rough estimate to provide the user some feedback but is not used for anything else to my knowledge. So you can ignore the run time estimate/remaining time estimate/percent progress and assume the job will run close to the cpu run time preference.


That will be helpful, thanks. (I had the target run-time set to 24 hours... That's where my issue came from, I've just set it back to the default.)

BTW
Umm, I don't know how this thread got led to schedulers, number of work assignments, and the observation that changing the target run-time seemingly doesn't effect tasks already in memory unless reloaded (i.e. by restarting BOINC).

Small question, does the average processing rate for Rosetta@home not take into account the fact that a longer running task on the same computer means more work has been done. It appear to be working on the belief that all Rosetta tasks, regardless of target run time, have a computational size of around 80,000GFLOPS.
ID: 95870 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 95873 - Posted: 2 May 2020, 21:52:19 UTC - in response to Message 95868.  

That may be the case on my main rig. Updating the target time setting didn't seem to affect tasks that had already started. Simply suspending and resuming does not work. I've never tried to restart BOINC and see if that made a difference.


You probably could also change the setting to NOT leave tasks in memory when suspended, and then suspending them (hopefully when very low time since last checkpoint), and resume them. ...then change the setting back.
Rosetta Moderator: Mod.Sense
ID: 95873 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95874 - Posted: 2 May 2020, 21:53:26 UTC - in response to Message 95873.  
Last modified: 2 May 2020, 22:12:16 UTC

That may be the case on my main rig. Updating the target time setting didn't seem to affect tasks that had already started. Simply suspending and resuming does not work. I've never tried to restart BOINC and see if that made a difference.


You probably could also change the setting to NOT leave tasks in memory when suspended, and then suspending them (hopefully when very low time since last checkpoint), and resume them. ...then change the setting back.



That's a good point, that may solve the issue. I'll try it on Ralph.
My previous target run-time was set to 24 hours, I've just updated it to the default of 1 hour (with 4 tasks in memory).
I've made sure that the venue is correct this time, and I made sure to update the project (with the new URL, too)
If, after 10 hours, the tasks don't finish, then I'll disable the "leave tasks in memory" setting.

Now that I think about it. Maybe the reason it worked in the past was due to that particular setting being somehow disabled on my old computer (tom's_old_unreliable)? Or it could be a false memory. I shall test this.
ID: 95874 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95876 - Posted: 2 May 2020, 22:00:42 UTC - in response to Message 95868.  

Sorry, I stand corrected. The cpu run time preference is written and read from init_data.xml. I thought this file gets updated if the preferences were updated but it may just get initialized at the start. I'm not sure though. But if it does just get initialized and never updated, then this would be consistent with what you are observing and the jobs will continue to run with the initial preference.

The job that ran 12+ hours instead of 8 was likely due to the specific task; by random chance it may have passed a filter and proceeded to run longer for the last model.
ID: 95876 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95879 - Posted: 2 May 2020, 22:16:48 UTC - in response to Message 95876.  
Last modified: 2 May 2020, 22:18:48 UTC

Sorry, I stand corrected. The cpu run time preference is written and read from init_data.xml. I thought this file gets updated if the preferences were updated but it may just get initialized at the start. I'm not sure though. But if it does just get initialized and never updated, then this would be consistent with what you are observing and the jobs will continue to run with the initial preference.

The job that ran 12+ hours instead of 8 was likely due to the specific task; by random chance it may have passed a filter and proceeded to run longer for the last model.


Thanks.
I'll test the CPU run-time preference a little more on Ralph.
My previous target run-time was set to 24 hours, I've just updated it to the default of 1 hour (with 4 tasks in memory).
If, after 10 hours, the tasks don't finish (I don't think they will), that means the CPU run-time preference has not been updated for those tasks.I'll then disable the "leave tasks in memory" setting. Suspend then resume those 4 tasks, and see if they end in a timely manner. If they do, that may suggest that init_data.xml does get updated, but only gets read when a task is loaded into memory.
I hope my logic is not completely off the wall. Brain... tired... need... sleep...
ID: 95879 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95880 - Posted: 2 May 2020, 22:36:16 UTC - in response to Message 95870.  


Small question, does the average processing rate for Rosetta@home not take into account the fact that a longer running task on the same computer means more work has been done. It appear to be working on the belief that all Rosetta tasks, regardless of target run time, have a computational size of around 80,000GFLOPS.


What is the average processing rate? And what is it used to calculate?

Related to your initial question, R@h credits are based on a rolling average of claimed credit/model that our server keeps track of for each job set. So you are awarded the average credit per model that previous results have reported back multiplied by the number of models you produced. The first job reported back for a specific job set is granted the claimed credit, but subsequent results are granted the average credit per model * the number of models produced. This works quite well for large job sets (like for protein structure prediction) because a job set may consist of thousands of jobs which eventually gives a good average credit per model value. But it can be a bit variable for small job sets, and if a job set consists of only one job, then you will be granted your claimed credit which is based on the cpu benchmark and run time.
ID: 95880 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1673
Credit: 17,596,129
RAC: 22,414
Message 95884 - Posted: 2 May 2020, 23:24:20 UTC - in response to Message 95880.  

What is the average processing rate? And what is it used to calculate?
It's basically an indicator of the compute performance for that host, for that particular application.
It was the replacement for the DCF (Duration Correction Factor). So it's used for runtime Estimates & also works in with the claiming of credit (under Credit New).

The problem with DCF was that it was very unstable and extremely variable, and on a project with more than 1 application or type of Work Unit it was even worse.
APR (Average Processing Rate) is per host and application version (or per host, app, resource type (eg CPU or GPU) in the case of anonymous platform) and tends to be less variable.

Job runtime estimation
It has a large overlap with the new credit system (which i don't think anyone really understands, i certainly don't).
Grant
Darwin NT
ID: 95884 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95886 - Posted: 2 May 2020, 23:38:14 UTC

Hmm, can anyone point me to the code on git where this is set and used? I'm still not understanding exactly what this is in context of the boinc code.
ID: 95886 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95893 - Posted: 3 May 2020, 3:16:01 UTC - in response to Message 95879.  


Thanks.
I'll test the CPU run-time preference a little more on Ralph.


It updates while running on my mac as I previously thought it would so I'm not sure what is happening with your host. A quick test is to update the background color of the graphics. When I update this preference on the web site and then click update on the client (so it communicates with the server to update the preferences) and then check the graphics, the background does change for a currently running job.
ID: 95893 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 389
Credit: 12,072,587
RAC: 8,261
Message 95912 - Posted: 3 May 2020, 9:51:33 UTC - in response to Message 95869.  

The estimated run times that the client provides can be off and irregular. But the job cache should hopefully now be more accurate given the deadlines and cache settings. Let us know if that is not the case with your current settings? It looks ok to me since .2 + .2 days of work = 9.6 hours and 10 work units = 10 hours with a cpu run time preference set at 1hour (the default).

The feeder part of the scheduler was updated to hopefully send a more reasonable number of jobs depending on the deadlines, cache settings, and existing cache, based on the cpu run time preference. This change will not affect the estimated run time information that the client provides which can be off. This estimate that the client displays is just a rough estimate to provide the user some feedback but is not used for anything else to my knowledge. So you can ignore the run time estimate/remaining time estimate/percent progress and assume the job will run close to the cpu run time preference.


Brilliant, thank you for all the hard work you’re putting in.
ID: 95912 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 389
Credit: 12,072,587
RAC: 8,261
Message 95913 - Posted: 3 May 2020, 9:56:28 UTC - in response to Message 95880.  


Small question, does the average processing rate for Rosetta@home not take into account the fact that a longer running task on the same computer means more work has been done. It appear to be working on the belief that all Rosetta tasks, regardless of target run time, have a computational size of around 80,000GFLOPS.


What is the average processing rate? And what is it used to calculate?

Related to your initial question, R@h credits are based on a rolling average of claimed credit/model that our server keeps track of for each job set. So you are awarded the average credit per model that previous results have reported back multiplied by the number of models you produced. The first job reported back for a specific job set is granted the claimed credit, but subsequent results are granted the average credit per model * the number of models produced. This works quite well for large job sets (like for protein structure prediction) because a job set may consist of thousands of jobs which eventually gives a good average credit per model value. But it can be a bit variable for small job sets, and if a job set consists of only one job, then you will be granted your claimed credit which is based on the cpu benchmark and run time.


That begs the question, how is the claimed credit calculated? That appears to be the driver for the whole process.
ID: 95913 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95942 - Posted: 3 May 2020, 19:06:20 UTC

The claimed credit is described here https://boinc.berkeley.edu/trac/wiki/CreditNew
ID: 95942 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95945 - Posted: 3 May 2020, 19:36:31 UTC

Tomcat, I'm looking into the details of the crediting system and will do my best to fix this issue.
ID: 95945 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95950 - Posted: 3 May 2020, 20:34:12 UTC - in response to Message 95893.  
Last modified: 3 May 2020, 20:50:36 UTC


Thanks.
I'll test the CPU run-time preference a little more on Ralph.


It updates while running on my mac as I previously thought it would so I'm not sure what is happening with your host. A quick test is to update the background color of the graphics. When I update this preference on the web site and then click update on the client (so it communicates with the server to update the preferences) and then check the graphics, the background does change for a currently running job.



Just did a little more testing in Ralph. It did work on my Mac, too. Well, it worked for one, three errored out (though probably for unrelated reasons, my Mac is an uncontrollable dumpster fire)...
These four tasks started with the target run-time set to 24 hours, then I updated them to run for 1 hour
https://ralph.bakerlab.org/result.php?resultid=5112642 (ended early, as expected)
https://ralph.bakerlab.org/result.php?resultid=5112572
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
exceeded elapsed time limit 35743.08 (500000.00G/13.99G)</message>
<stderr_txt>
command: rosetta_4.20_x86_64-apple-darwin @rb_04_25_22832_22285_ab_t000__robetta_FLAGS -in::file::fasta t000_.fasta -constraints::cst_file t000_.fasta.CB.cst -constraints:cst_weight 5.0 -constraints::cst_fa_file t000_.fasta.MIN.cst -constraints:cst_fa_weight 5.0 -in:file:boinc_wu_zip rb_04_25_22832_22285_ab_t000__robetta.zip -frag3 rb_04_25_22832_22285_ab_t000__robetta.200.3mers.index.gz -fragA rb_04_25_22832_22285_ab_t000__robetta.200.7mers.index.gz -fragB rb_04_25_22832_22285_ab_t000__robetta.200.3mers.index.gz -nstruct 10000 -cpu_run_time 3600 -boinc:max_nstruct 5000 -checkpoint_interval 120 -mute all -database minirosetta_database -in::file::zip minirosetta_database.zip -boinc::watchdog -boinc::cpu_run_timeout 36000 -run::rng mt19937 -constant_seed -jran 3967979
Extracting in project directory: database_357d5d93529_n_methyl.zip
Using database: database_357d5d93529_n_methyl/minirosetta_database



</stderr_txt>
]]>


https://ralph.bakerlab.org/result.php?resultid=5112573
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
exceeded elapsed time limit 35743.08 (500000.00G/13.99G)</message>
<stderr_txt>
command: rosetta_4.20_x86_64-apple-darwin @rb_04_25_22832_22285_ab_t000__robetta_FLAGS -in::file::fasta t000_.fasta -constraints::cst_file t000_.fasta.CB.cst -constraints:cst_weight 5.0 -constraints::cst_fa_file t000_.fasta.MIN.cst -constraints:cst_fa_weight 5.0 -in:file:boinc_wu_zip rb_04_25_22832_22285_ab_t000__robetta.zip -frag3 rb_04_25_22832_22285_ab_t000__robetta.200.3mers.index.gz -fragA rb_04_25_22832_22285_ab_t000__robetta.200.7mers.index.gz -fragB rb_04_25_22832_22285_ab_t000__robetta.200.4mers.index.gz -nstruct 10000 -cpu_run_time 3600 -boinc:max_nstruct 5000 -checkpoint_interval 120 -mute all -database minirosetta_database -in::file::zip minirosetta_database.zip -boinc::watchdog -boinc::cpu_run_timeout 36000 -run::rng mt19937 -constant_seed -jran 3967967
Using database: database_357d5d93529_n_methyl/minirosetta_database

Crashed executable name: rosetta_4.20_x86_64-apple-darwin
built using BOINC library version 7.9.0
Machine type Intel x86-64h Haswell (64-bit executable)
System version: Macintosh OS 10.15.1 build 19B88
Sun May  3 02:23:53 2020

atos cannot load symbols for the file rosetta_4.20_x86_64-apple-darwin for architecture x86_64.
0   rosetta_4.20_x86_64-apple-darwin    0x000000011139e856  
1   rosetta_4.20_x86_64-apple-darwin    0x000000011138a2c6  
SIGPIPE: write on a pipe with no reader
2   libsystem_pthread.dylib             0x00007fff70ccbd36  
SIGPIPE: write on a pipe with no reader
3   libsystem_pthread.dylib             0x00007fff70cc858f  
SIGPIPE: write on a pipe with no reader

Thread 1 crashed with X86 Thread State (64-bit):
  rax: 0x0100001f  rbx: 0x00000003  rcx: 0x70000ad2ff38  rdx: 0x00000028
  rdi: 0x70000ad2ffa8  rsi: 0x00000003  rbp: 0x70000ad2ff90  rsp: 0x70000ad2ff38
   r8: 0x00001703   r9: 0x00000000  r10: 0x000009c8  r11: 0x00000206
  r12: 0x00000003  r13: 0x000009c8  r14: 0x70000ad2ffa8  r15: 0x00000028
  rip: 0x7fff70c08166  rfl: 0x00000206

Binary Images Description:
       0x10d051000 -        0x112086fff /Library/Application Support/BOINC Data/slots/3/../../projects/ralph.bakerlab.org/rosetta_4.20_x86_64-apple-darwin
    0x7fff3537b000 -     0x7fff3537bfff /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    0x7fff35393000 -     0x7fff359fefff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
    0x7fff359ff000 -     0x7fff35c67fff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
    0x7fff35c68000 -     0x7fff35f57fff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
    0x7fff35f59000 -     0x7fff362fefff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
    0x7fff362ff000 -     0x7fff36314fff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
    0x7fff36315000 -     0x7fff3631afff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
    0x7fff3631b000 -     0x7fff3638bfff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib
    0x7fff3638c000 -     0x7fff3639efff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
    0x7fff3639f000 -     0x7fff36578fff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
    0x7fff36579000 -     0x7fff36634fff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
    0x7fff36635000 -     0x7fff36635fff /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
    0x7fff367da000 -     0x7fff37592fff /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
    0x7fff375e2000 -     0x7fff375e2fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
    0x7fff375e3000 -     0x7fff3764efff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
    0x7fff376e7000 -     0x7fff37725fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
    0x7fff37780000 -     0x7fff377affff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATSUI.framework/Versions/A/ATSUI
    0x7fff377b0000 -     0x7fff377b4fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSyncLegacy.framework/Versions/A/ColorSyncLegacy
    0x7fff3784f000 -     0x7fff378a5fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
    0x7fff378a6000 -     0x7fff378b4fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
    0x7fff378b5000 -     0x7fff378fafff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
    0x7fff378fb000 -     0x7fff37905fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
    0x7fff37906000 -     0x7fff37913fff /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
    0x7fff37914000 -     0x7fff379f4fff /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
    0x7fff379f6000 -     0x7fff379f6fff /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
    0x7fff37d6c000 -     0x7fff380e3fff /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
    0x7fff38162000 -     0x7fff38456fff /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
    0x7fff384a1000 -     0x7fff384a7fff /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
    0x7fff38642000 -     0x7fff38642fff /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
    0x7fff38650000 -     0x7fff3883bfff /System/Library/Frameworks/ColorSync.framework/Versions/A/ColorSync
    0x7fff38b25000 -     0x7fff39034fff /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
    0x7fff39087000 -     0x7fff390befff /System/Library/Frameworks/CoreBluetooth.framework/Versions/A/CoreBluetooth
    0x7fff390bf000 -     0x7fff3949dfff /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
    0x7fff3949e000 -     0x7fff395aafff /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
    0x7fff395ab000 -     0x7fff39a2afff /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fff39a2c000 -     0x7fff3a0a4fff /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
    0x7fff3a0b2000 -     0x7fff3a40ffff /System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage
    0x7fff3a998000 -     0x7fff3a998fff /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
    0x7fff3a999000 -     0x7fff3aa1efff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
    0x7fff3aa1f000 -     0x7fff3ad00fff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
    0x7fff3ad01000 -     0x7fff3ad4efff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
    0x7fff3ad4f000 -     0x7fff3ad57fff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
    0x7fff3ad58000 -     0x7fff3af91fff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
    0x7fff3af92000 -     0x7fff3b02afff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
    0x7fff3b02b000 -     0x7fff3b058fff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
    0x7fff3b059000 -     0x7fff3b0c0fff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
    0x7fff3b0c1000 -     0x7fff3b0e5fff /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
    0x7fff3b40d000 -     0x7fff3b5c1fff /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
    0x7fff3b5c2000 -     0x7fff3b606fff /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
    0x7fff3b607000 -     0x7fff3b694fff /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN
    0x7fff3b936000 -     0x7fff3b93cfff /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
    0x7fff3bc6e000 -     0x7fff3c035fff /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
    0x7fff3c037000 -     0x7fff3c05efff /System/Library/Frameworks/GLKit.framework/Versions/A/GLKit
    0x7fff3c064000 -     0x7fff3c0a1fff /System/Library/Frameworks/GLUT.framework/Versions/A/GLUT
    0x7fff3c0a2000 -     0x7fff3c0f2fff /System/Library/Frameworks/GSS.framework/Versions/A/GSS
    0x7fff3c22c000 -     0x7fff3c344fff /System/Library/Frameworks/IOBluetooth.framework/Versions/A/IOBluetooth
    0x7fff3c3ab000 -     0x7fff3c44efff /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
    0x7fff3c450000 -     0x7fff3c460fff /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
    0x7fff3c4d7000 -     0x7fff3c633fff /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
    0x7fff3c634000 -     0x7fff3c637fff /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
    0x7fff3c638000 -     0x7fff3c6f2fff /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
    0x7fff3c6f3000 -     0x7fff3c717fff /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
    0x7fff3c995000 -     0x7fff3c9affff /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
    0x7fff3c9b0000 -     0x7fff3c9b1fff /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
    0x7fff3c9b2000 -     0x7fff3c9fbfff /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
    0x7fff3de20000 -     0x7fff3de32fff /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
    0x7fff3de33000 -     0x7fff3de33fff /System/Library/Frameworks/Kerberos.framework/Versions/A/Libraries/libHeimdalProxy.dylib
    0x7fff3e9e1000 -     0x7fff3eaa4fff /System/Library/Frameworks/Metal.framework/Versions/A/Metal
    0x7fff3eac1000 -     0x7fff3eafdfff /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSCore.framework/Versions/A/MPSCore
    0x7fff3eafe000 -     0x7fff3eb84fff /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSImage.framework/Versions/A/MPSImage
    0x7fff3eb85000 -     0x7fff3eba9fff /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSMatrix.framework/Versions/A/MPSMatrix
    0x7fff3ebaa000 -     0x7fff3ebbffff /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSNDArray.framework/Versions/A/MPSNDArray
    0x7fff3ebc0000 -     0x7fff3ed1ffff /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSNeuralNetwork.framework/Versions/A/MPSNeuralNetwork
    0x7fff3ed20000 -     0x7fff3ed6efff /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSRayIntersector.framework/Versions/A/MPSRayIntersector
    0x7fff3ed6f000 -     0x7fff3ed70fff /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/MetalPerformanceShaders
    0x7fff3ed7e000 -     0x7fff3fc30fff /System/Library/Frameworks/ModelIO.framework/Versions/A/ModelIO
    0x7fff3fcb6000 -     0x7fff3fcc2fff /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
    0x7fff3fcc3000 -     0x7fff3fe06fff /System/Library/Frameworks/Network.framework/Versions/A/Network
    0x7fff4282f000 -     0x7fff42887fff /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
    0x7fff42888000 -     0x7fff428a4fff /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
    0x7fff428a5000 -     0x7fff428b0fff /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
    0x7fff4320b000 -     0x7fff4320dfff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
    0x7fff4320e000 -     0x7fff43213fff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreFSCache.dylib
    0x7fff43214000 -     0x7fff43218fff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
    0x7fff43219000 -     0x7fff43221fff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
    0x7fff43222000 -     0x7fff4322cfff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
    0x7fff4322d000 -     0x7fff43262fff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
    0x7fff433f6000 -     0x7fff43432fff /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
    0x7fff43e62000 -     0x7fff43e71fff /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
    0x7fff44e2b000 -     0x7fff450a9fff /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
    0x7fff45bf6000 -     0x7fff45f3cfff /System/Library/Frameworks/Security.framework/Versions/A/Security
    0x7fff45f3d000 -     0x7fff45fc5fff /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
    0x7fff4601f000 -     0x7fff46023fff /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
    0x7fff46db0000 -     0x7fff46e1afff /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
    0x7fff4ab76000 -     0x7fff4ac3afff /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
    0x7fff4bd16000 -     0x7fff4bd17fff /System/Library/PrivateFrameworks/AggregateDictionary.framework/Versions/A/AggregateDictionary
    0x7fff4c72e000 -     0x7fff4c752fff /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Apple80211
    0x7fff4c888000 -     0x7fff4c897fff /System/Library/PrivateFrameworks/AppleFSCompression.framework/Versions/A/AppleFSCompression
    0x7fff4c996000 -     0x7fff4c9a1fff /System/Library/PrivateFrameworks/AppleIDAuthSupport.framework/Versions/A/AppleIDAuthSupport
    0x7fff4c9e3000 -     0x7fff4ca2bfff /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
    0x7fff4ce09000 -     0x7fff4ce2bfff /System/Library/PrivateFrameworks/AppleSauce.framework/Versions/A/AppleSauce
    0x7fff4ceeb000 -     0x7fff4ceeefff /System/Library/PrivateFrameworks/AppleSystemInfo.framework/Versions/A/AppleSystemInfo
    0x7fff4cf88000 -     0x7fff4cf97fff /System/Library/PrivateFrameworks/AssertionServices.framework/Versions/A/AssertionServices
    0x7fff4db28000 -     0x7fff4dd65fff /System/Library/PrivateFrameworks/AudioToolboxCore.framework/Versions/A/AudioToolboxCore
    0x7fff4dd66000 -     0x7fff4de7ffff /System/Library/PrivateFrameworks/AuthKit.framework/Versions/A/AuthKit
    0x7fff4e03a000 -     0x7fff4e043fff /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
    0x7fff4e044000 -     0x7fff4e0e5fff /System/Library/PrivateFrameworks/Backup.framework/Versions/A/Backup
    0x7fff4e0e6000 -     0x7fff4e167fff /System/Library/PrivateFrameworks/BaseBoard.framework/Versions/A/BaseBoard
    0x7fff4e269000 -     0x7fff4e2a5fff /System/Library/PrivateFrameworks/Bom.framework/Versions/A/Bom
    0x7fff4fcf2000 -     0x7fff4fd03fff /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
    0x7fff4fd17000 -     0x7fff4fd2efff /System/Library/PrivateFrameworks/CommonUtilities.framework/Versions/A/CommonUtilities
    0x7fff5084e000 -     0x7fff5086dfff /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/CoreAnalytics
    0x7fff50dbb000 -     0x7fff50dcbfff /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
    0x7fff5141c000 -     0x7fff51486fff /System/Library/PrivateFrameworks/CoreNLP.framework/Versions/A/CoreNLP
    0x7fff518f5000 -     0x7fff518fdfff /System/Library/PrivateFrameworks/CorePhoneNumbers.framework/Versions/A/CorePhoneNumbers
    0x7fff52049000 -     0x7fff5206cfff /System/Library/PrivateFrameworks/CoreSVG.framework/Versions/A/CoreSVG
    0x7fff520a1000 -     0x7fff520cffff /System/Library/PrivateFrameworks/CoreServicesStore.framework/Versions/A/CoreServicesStore
    0x7fff526ef000 -     0x7fff5281bfff /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
    0x7fff5281c000 -     0x7fff529b7fff /System/Library/PrivateFrameworks/CoreUtils.framework/Versions/A/CoreUtils
    0x7fff52aec000 -     0x7fff52afffff /System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
    0x7fff52d6b000 -     0x7fff52d7dfff /System/Library/PrivateFrameworks/DFRFoundation.framework/Versions/A/DFRFoundation
    0x7fff52d7e000 -     0x7fff52d83fff /System/Library/PrivateFrameworks/DSExternalDisplay.framework/Versions/A/DSExternalDisplay
    0x7fff52dec000 -     0x7fff52e67fff /System/Library/PrivateFrameworks/DataDetectorsCore.framework/Versions/A/DataDetectorsCore
    0x7fff52ef2000 -     0x7fff5304dfff /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
    0x7fff54881000 -     0x7fff54c9cfff /System/Library/PrivateFrameworks/FaceCore.framework/Versions/A/FaceCore
    0x7fff55302000 -     0x7fff55439fff /System/Library/PrivateFrameworks/FontServices.framework/libFontParser.dylib
    0x7fff554d3000 -     0x7fff554e3fff /System/Library/PrivateFrameworks/FontServices.framework/libhvf.dylib
    0x7fff5a04f000 -     0x7fff5a055fff /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
    0x7fff5b4b4000 -     0x7fff5b4c2fff /System/Library/PrivateFrameworks/GraphVisualizer.framework/Versions/A/GraphVisualizer
    0x7fff5b655000 -     0x7fff5b713fff /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
    0x7fff5d84a000 -     0x7fff5d852fff /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
    0x7fff5d855000 -     0x7fff5d86bfff /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
    0x7fff5dbf5000 -     0x7fff5dc40fff /System/Library/PrivateFrameworks/IconServices.framework/Versions/A/IconServices
    0x7fff5ddfd000 -     0x7fff5de03fff /System/Library/PrivateFrameworks/InternationalSupport.framework/Versions/A/InternationalSupport
    0x7fff5e044000 -     0x7fff5e064fff /System/Library/PrivateFrameworks/KeychainCircle.framework/Versions/A/KeychainCircle
    0x7fff5e1bb000 -     0x7fff5e289fff /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
    0x7fff5e28a000 -     0x7fff5e2d2fff /System/Library/PrivateFrameworks/Lexicon.framework/Versions/A/Lexicon
    0x7fff5e2d9000 -     0x7fff5e2ddfff /System/Library/PrivateFrameworks/LinguisticData.framework/Versions/A/LinguisticData
    0x7fff5f62e000 -     0x7fff5f67afff /System/Library/PrivateFrameworks/MetadataUtilities.framework/Versions/A/MetadataUtilities
    0x7fff5f67b000 -     0x7fff5f749fff /System/Library/PrivateFrameworks/MetalTools.framework/Versions/A/MetalTools
    0x7fff5f979000 -     0x7fff5f997fff /System/Library/PrivateFrameworks/MobileKeyBag.framework/Versions/A/MobileKeyBag
    0x7fff5fbfe000 -     0x7fff5fc2cfff /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
    0x7fff600a1000 -     0x7fff600abfff /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
    0x7fff60a99000 -     0x7fff60ae5fff /System/Library/PrivateFrameworks/OTSVG.framework/Versions/A/OTSVG
    0x7fff61c74000 -     0x7fff61c7ffff /System/Library/PrivateFrameworks/PerformanceAnalysis.framework/Versions/A/PerformanceAnalysis
    0x7fff61c80000 -     0x7fff61ca8fff /System/Library/PrivateFrameworks/PersistentConnection.framework/Versions/A/PersistentConnection
    0x7fff645c8000 -     0x7fff645e1fff /System/Library/PrivateFrameworks/ProtocolBuffer.framework/Versions/A/ProtocolBuffer
    0x7fff649e6000 -     0x7fff64a0ffff /System/Library/PrivateFrameworks/RemoteViewServices.framework/Versions/A/RemoteViewServices
    0x7fff64b74000 -     0x7fff64baffff /System/Library/PrivateFrameworks/RunningBoardServices.framework/Versions/A/RunningBoardServices
    0x7fff6660e000 -     0x7fff66732fff /System/Library/PrivateFrameworks/Sharing.framework/Versions/A/Sharing
    0x7fff67766000 -     0x7fff67a5efff /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
    0x7fff682a6000 -     0x7fff682b4fff /System/Library/PrivateFrameworks/SpeechRecognitionCore.framework/Versions/A/SpeechRecognitionCore
    0x7fff68d95000 -     0x7fff68da5fff /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
    0x7fff69297000 -     0x7fff6935efff /System/Library/PrivateFrameworks/TextureIO.framework/Versions/A/TextureIO
    0x7fff6a1b5000 -     0x7fff6a40efff /System/Library/PrivateFrameworks/UIFoundation.framework/Versions/A/UIFoundation
    0x7fff6c060000 -     0x7fff6c061fff /System/Library/PrivateFrameworks/WatchdogClient.framework/Versions/A/WatchdogClient
    0x7fff6cc38000 -     0x7fff6cc3bfff /System/Library/PrivateFrameworks/XCTTargetBootstrap.framework/Versions/A/XCTTargetBootstrap
    0x7fff6ccb4000 -     0x7fff6ccc2fff /System/Library/PrivateFrameworks/caulk.framework/Versions/A/caulk
    0x7fff6d002000 -     0x7fff6d004fff /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
    0x7fff6d337000 -     0x7fff6d36cfff /usr/lib/libAudioToolboxUtility.dylib
    0x7fff6d373000 -     0x7fff6d3a8fff /usr/lib/libCRFSuite.dylib
    0x7fff6d3ab000 -     0x7fff6d3b5fff /usr/lib/libChineseTokenizer.dylib
    0x7fff6d442000 -     0x7fff6d444fff /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fff6d489000 -     0x7fff6d640fff /usr/lib/libFosl_dynamic.dylib
    0x7fff6d667000 -     0x7fff6d66dfff /usr/lib/libIOReport.dylib
    0x7fff6d784000 -     0x7fff6d7a3fff /usr/lib/libMobileGestalt.dylib
    0x7fff6d90a000 -     0x7fff6d90bfff /usr/lib/libSystem.B.dylib
    0x7fff6d99b000 -     0x7fff6d99cfff /usr/lib/libThaiTokenizer.dylib
    0x7fff6d9b4000 -     0x7fff6d9cafff /usr/lib/libapple_nghttp2.dylib
    0x7fff6d9ff000 -     0x7fff6da71fff /usr/lib/libarchive.2.dylib
    0x7fff6da72000 -     0x7fff6db08fff /usr/lib/libate.dylib
    0x7fff6db0c000 -     0x7fff6db0cfff /usr/lib/libauto.dylib
    0x7fff6dbca000 -     0x7fff6dbdafff /usr/lib/libbsm.0.dylib
    0x7fff6dbdb000 -     0x7fff6dbe7fff /usr/lib/libbz2.1.0.dylib
    0x7fff6dbe8000 -     0x7fff6dc3bfff /usr/lib/libc++.1.dylib
    0x7fff6dc3c000 -     0x7fff6dc50fff /usr/lib/libc++abi.dylib
    0x7fff6dc51000 -     0x7fff6dc51fff /usr/lib/libcharset.1.dylib
    0x7fff6dc52000 -     0x7fff6dc63fff /usr/lib/libcmph.dylib
    0x7fff6dc64000 -     0x7fff6dc7bfff /usr/lib/libcompression.dylib
    0x7fff6df48000 -     0x7fff6df5efff /usr/lib/libcoretls.dylib
    0x7fff6df5f000 -     0x7fff6df60fff /usr/lib/libcoretls_cfhelpers.dylib
    0x7fff6e51e000 -     0x7fff6e57dfff /usr/lib/libcups.2.dylib
    0x7fff6e689000 -     0x7fff6e689fff /usr/lib/libenergytrace.dylib
    0x7fff6e68a000 -     0x7fff6e6a3fff /usr/lib/libexpat.1.dylib
    0x7fff6e6b1000 -     0x7fff6e6b3fff /usr/lib/libfakelink.dylib
    0x7fff6e6c2000 -     0x7fff6e6c7fff /usr/lib/libgermantok.dylib
    0x7fff6e6c8000 -     0x7fff6e6d1fff /usr/lib/libheimdal-asn1.dylib
    0x7fff6e6d2000 -     0x7fff6e7c2fff /usr/lib/libiconv.2.dylib
    0x7fff6e7c3000 -     0x7fff6ea1bfff /usr/lib/libicucore.A.dylib
    0x7fff6ea35000 -     0x7fff6ea36fff /usr/lib/liblangid.dylib
    0x7fff6ea37000 -     0x7fff6ea4ffff /usr/lib/liblzma.5.dylib
    0x7fff6ea67000 -     0x7fff6eb0efff /usr/lib/libmecab.dylib
    0x7fff6eb0f000 -     0x7fff6ed6ffff /usr/lib/libmecabra.dylib
    0x7fff6f0db000 -     0x7fff6f10afff /usr/lib/libncurses.5.4.dylib
    0x7fff6f239000 -     0x7fff6f6abfff /usr/lib/libnetwork.dylib
    0x7fff6f74a000 -     0x7fff6f77bfff /usr/lib/libobjc.A.dylib
    0x7fff6f78e000 -     0x7fff6f792fff /usr/lib/libpam.2.dylib
    0x7fff6f795000 -     0x7fff6f7c8fff /usr/lib/libpcap.A.dylib
    0x7fff6f84a000 -     0x7fff6f862fff /usr/lib/libresolv.9.dylib
    0x7fff6f8be000 -     0x7fff6faabfff /usr/lib/libsqlite3.dylib
    0x7fff6fcfc000 -     0x7fff6fcfffff /usr/lib/libutil.dylib
    0x7fff6fd00000 -     0x7fff6fd0dfff /usr/lib/libxar.1.dylib
    0x7fff6fd13000 -     0x7fff6fdf5fff /usr/lib/libxml2.2.dylib
    0x7fff6fdf9000 -     0x7fff6fe21fff /usr/lib/libxslt.1.dylib
    0x7fff6fe22000 -     0x7fff6fe34fff /usr/lib/libz.1.dylib
    0x7fff7089a000 -     0x7fff7089ffff /usr/lib/system/libcache.dylib
    0x7fff708a0000 -     0x7fff708abfff /usr/lib/system/libcommonCrypto.dylib
    0x7fff708ac000 -     0x7fff708b3fff /usr/lib/system/libcompiler_rt.dylib
    0x7fff708b4000 -     0x7fff708bdfff /usr/lib/system/libcopyfile.dylib
    0x7fff708be000 -     0x7fff70955fff /usr/lib/system/libcorecrypto.dylib
    0x7fff70a6c000 -     0x7fff70aadfff /usr/lib/system/libdispatch.dylib
    0x7fff70aae000 -     0x7fff70ae3fff /usr/lib/system/libdyld.dylib
    0x7fff70ae4000 -     0x7fff70ae4fff /usr/lib/system/libkeymgr.dylib
    0x7fff70ae5000 -     0x7fff70af1fff /usr/lib/system/libkxld.dylib
    0x7fff70af2000 -     0x7fff70af2fff /usr/lib/system/liblaunch.dylib
    0x7fff70af3000 -     0x7fff70af8fff /usr/lib/system/libmacho.dylib
    0x7fff70af9000 -     0x7fff70afbfff /usr/lib/system/libquarantine.dylib
    0x7fff70afc000 -     0x7fff70afdfff /usr/lib/system/libremovefile.dylib
    0x7fff70afe000 -     0x7fff70b15fff /usr/lib/system/libsystem_asl.dylib
    0x7fff70b16000 -     0x7fff70b16fff /usr/lib/system/libsystem_blocks.dylib
    0x7fff70b17000 -     0x7fff70b9efff /usr/lib/system/libsystem_c.dylib
    0x7fff70b9f000 -     0x7fff70ba2fff /usr/lib/system/libsystem_configuration.dylib
    0x7fff70ba3000 -     0x7fff70ba6fff /usr/lib/system/libsystem_coreservices.dylib
    0x7fff70ba7000 -     0x7fff70baefff /usr/lib/system/libsystem_darwin.dylib
    0x7fff70baf000 -     0x7fff70bb6fff /usr/lib/system/libsystem_dnssd.dylib
    0x7fff70bb7000 -     0x7fff70bb8fff /usr/lib/system/libsystem_featureflags.dylib
    0x7fff70bb9000 -     0x7fff70c06fff /usr/lib/system/libsystem_info.dylib
    0x7fff70c07000 -     0x7fff70c33fff /usr/lib/system/libsystem_kernel.dylib
    0x7fff70c34000 -     0x7fff70c7bfff /usr/lib/system/libsystem_m.dylib
    0x7fff70c7c000 -     0x7fff70ca3fff /usr/lib/system/libsystem_malloc.dylib
    0x7fff70ca4000 -     0x7fff70cb1fff /usr/lib/system/libsystem_networkextension.dylib
    0x7fff70cb2000 -     0x7fff70cbbfff /usr/lib/system/libsystem_notify.dylib
    0x7fff70cbc000 -     0x7fff70cc5fff /usr/lib/system/libsystem_platform.dylib
    0x7fff70cc6000 -     0x7fff70cd0fff /usr/lib/system/libsystem_pthread.dylib
    0x7fff70cd1000 -     0x7fff70cd5fff /usr/lib/system/libsystem_sandbox.dylib
    0x7fff70cd6000 -     0x7fff70cd8fff /usr/lib/system/libsystem_secinit.dylib
    0x7fff70cd9000 -     0x7fff70ce0fff /usr/lib/system/libsystem_symptoms.dylib
    0x7fff70ce1000 -     0x7fff70cf7fff /usr/lib/system/libsystem_trace.dylib
    0x7fff70cf9000 -     0x7fff70cfefff /usr/lib/system/libunwind.dylib
    0x7fff70cff000 -     0x7fff70d33fff /usr/lib/system/libxpc.dylib


</stderr_txt>
]]>


https://ralph.bakerlab.org/result.php?resultid=5112859
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
exceeded elapsed time limit 35743.08 (500000.00G/13.99G)</message>
<stderr_txt>
command: rosetta_4.20_x86_64-apple-darwin @rb_04_25_22832_22285_ab_t000__robetta_FLAGS -in::file::fasta t000_.fasta -constraints::cst_file t000_.fasta.CB.cst -constraints:cst_weight 5.0 -constraints::cst_fa_file t000_.fasta.MIN.cst -constraints:cst_fa_weight 5.0 -in:file:boinc_wu_zip rb_04_25_22832_22285_ab_t000__robetta.zip -frag3 rb_04_25_22832_22285_ab_t000__robetta.200.3mers.index.gz -fragA rb_04_25_22832_22285_ab_t000__robetta.200.8mers.index.gz -fragB rb_04_25_22832_22285_ab_t000__robetta.200.10mers.index.gz -nstruct 10000 -cpu_run_time 3600 -boinc:max_nstruct 5000 -checkpoint_interval 120 -mute all -database minirosetta_database -in::file::zip minirosetta_database.zip -boinc::watchdog -boinc::cpu_run_timeout 36000 -run::rng mt19937 -constant_seed -jran 3967824
Using database: database_357d5d93529_n_methyl/minirosetta_database

</stderr_txt>
]]>


For whatever reason, it doesn't seem to be working in Windows 10. The target runtime was set to the default when some of my tasks started (can't tell because they all ran for 1 hour or so). I updated the target runtime to 12 hours when they were partially complete (definately before 50%, but I don't remember the percentage), and they still ran for 1 hour (might be an edge case, though, since the target time was set so low)
I've started 3 Ralph tasks on my windows machine with the target runtime set to 12 hours. They've been running for 6 hours. I will now update the target runtime to 1 hour and see if they end early, just like on my Mac. I will update the colour to black and see if it changes...
ID: 95950 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95951 - Posted: 3 May 2020, 20:41:20 UTC - in response to Message 95950.  
Last modified: 3 May 2020, 20:43:16 UTC

Update, the colour did change. Still waiting for the tasks to finish early.
ID: 95951 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95952 - Posted: 3 May 2020, 20:44:24 UTC - in response to Message 95945.  

Tomcat, I'm looking into the details of the crediting system and will do my best to fix this issue.

Thanks.
ID: 95952 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tomcat雄猫

Send message
Joined: 20 Dec 14
Posts: 180
Credit: 5,386,173
RAC: 0
Message 95971 - Posted: 4 May 2020, 1:23:27 UTC - in response to Message 95952.  
Last modified: 4 May 2020, 1:25:03 UTC

Oh dear, same errors on my main rig... All three of them
Well, I'll stop touching the target runtime setting when there are tasks running, then.
https://ralph.bakerlab.org/result.php?resultid=5125819
https://ralph.bakerlab.org/result.php?resultid=5125818
https://ralph.bakerlab.org/result.php?resultid=5125816
<core_client_version>7.16.5</core_client_version>
<![CDATA[
<message>
exceeded elapsed time limit 38822.94 (500000.00G/12.88G)</message>
<stderr_txt>
command: projects/ralph.bakerlab.org/rosetta_4.20_windows_x86_64.exe -run:protocol jd2_scripting @flags_rb_04_25_22864_22281__t000__0_C1_robetta -silent_gz -mute all -out:file:silent default.out -in:file:boinc_wu_zip input_rb_04_25_22864_22281__t000__0_C1_robetta.zip -nstruct 10000 -cpu_run_time 3600 -boinc:max_nstruct 5000 -checkpoint_interval 120 -database minirosetta_database -in::file::zip minirosetta_database.zip -boinc::watchdog -boinc::cpu_run_timeout 36000 -run::rng mt19937 -constant_seed -jran 3957371
Using database: database_357d5d93529_n_methylminirosetta_database


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFB4B870AA2

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.9.0


Dump Timestamp    : 05/03/20 21:17:18
Install Directory : C:Program FilesBOINC
Data Directory    : A:BOINC
Project Symstore  : https://boinc.bakerlab.org/rosetta/symstore
LoadLibraryA( A:BOINCdbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( A:BOINCsymsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( A:BOINCsrcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( A:BOINCversion.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: A:BOINCslots2;A:BOINCprojectsralph.bakerlab.org;srv*A:BOINCprojectsralph.bakerlab.orgsymbols*http://msdl.microsoft.com/download/symbols;srv*A:BOINCprojectsralph.bakerlab.orgsymbols*https://boinc.bakerlab.org/rosetta/symstore


ModLoad: 0000000005420000 00000000057ef000 A:BOINCprojectsralph.bakerlab.orgrosetta_4.20_windows_x86_64.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : C:cygwin64homeboinc4.17RosettamainsourceideVisualStudiox64BoincReleaserosetta_4.20_windows_x86_64.pdb

ModLoad: 000000004dd60000 00000000001f0000 C:windowsSYSTEM32ntdll.dll (6.2.18362.778) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.18362.329 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.329

ModLoad: 000000004be10000 00000000000b2000 C:windowsSystem32KERNEL32.DLL (6.2.18362.778) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.18362.329 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.329

ModLoad: 000000004b7b0000 00000000002a3000 C:windowsSystem32KERNELBASE.dll (6.2.18362.778) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.18362.329 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.329

ModLoad: 000000004cca0000 000000000006f000 C:windowsSystem32WS2_32.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : ws2_32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004c8f0000 0000000000120000 C:windowsSystem32RPCRT4.dll (6.2.18362.628) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004bed0000 0000000000194000 C:windowsSystem32USER32.dll (6.2.18362.778) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004bc90000 0000000000021000 C:windowsSystem32win32u.dll (6.2.18362.778) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.18362.778 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.778

ModLoad: 000000004ce60000 0000000000026000 C:windowsSystem32GDI32.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004acf0000 0000000000194000 C:windowsSystem32gdi32full.dll (6.2.18362.778) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.18362.778 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.778

ModLoad: 000000004aee0000 000000000009e000 C:windowsSystem32msvcp_win.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 000000004ba60000 00000000000fa000 C:windowsSystem32ucrtbase.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 000000004c500000 00000000000a3000 C:windowsSystem32ADVAPI32.dll (6.2.18362.752) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004ca10000 000000000009e000 C:windowsSystem32msvcrt.dll (7.0.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.18362.1

ModLoad: 000000004cef0000 0000000000097000 C:windowsSystem32sechost.dll (6.2.18362.693) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004dbe0000 000000000002e000 C:windowsSystem32IMM32.DLL (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 000000004aca0000 0000000000011000 C:windowsSystem32kernel.appcore.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000049d30000 0000000000031000 C:windowsSYSTEM32ntmarta.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntmarta.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004a600000 000000000000c000 C:windowsSYSTEM32CRYPTBASE.DLL (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000004bc10000 0000000000080000 C:windowsSystem32bcryptPrimitives.dll (6.2.18362.295) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.18362.295 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.295

ModLoad: 0000000036920000 00000000001f4000 C:windowsSYSTEM32dbghelp.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000043c40000 000000000000a000 C:windowsSYSTEM32version.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 101664, Write: 21777, Other 38827

- I/O Transfers Counters -
Read: 405703674, Write: 84484832, Other 30422

- Paged Pool Usage -
QuotaPagedPoolUsage: 318568, QuotaPeakPagedPoolUsage: 318744
QuotaNonPagedPoolUsage: 21480, QuotaPeakNonPagedPoolUsage: 22584

- Virtual Memory Usage -
VirtualSize: 885653504, PeakVirtualSize: 1529880576

- Pagefile Usage -
PagefileUsage: 885653504, PeakPagefileUsage: 914309120

- Working Set Size -
WorkingSetSize: 626642944, PeakWorkingSetSize: 891756544, PageFaultCount: 58434046

*** Dump of thread ID 2500 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFB4B870AA2

- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=0000000008fb8060 rdx=000000005063f190 rsi=0000000000000000 rdi=0000000000000000
r8=000000005063f190 r9=0000000008fb8050 r10=0000000000000001 r11=0000000000000fff r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000004b870aa2 rsp=000000005063f168 rbp=0000000000000000
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
5063f160 058e6f1e 00000001 5063f190 5063f190 08fb8050 KERNELBASE!DebugBreak+0x0 
5063f5a0 058e7f10 00000000 00000000 00000000 00000000 rosetta_4.20_windows_x86_64!xmlParserInputRead+0x0 
5063f800 4be27bd4 00000000 00000000 00000000 00000000 rosetta_4.20_windows_x86_64!xmlParserInputRead+0x0 
5063f830 4ddcce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
5063f8b0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 32763 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Unknown, , Kernel Time: 30.000000, User Time: 156250.000000, Wait Time: 723142272.000000

- Registers -
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000000000000 rsp=0000000000000000 rbp=0000000000000000
cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000

- Callstack -
ChildEBP RetAddr  Args to Child
(-nosymbols- PC == 0)
00000000 00000000 00000000 00000000 00000000 00000000 !+0x0 

*** Dump of thread ID 30810432 (state: Unknown): ***

- Information -
Status: Base Priority: Normal, Priority: Unknown, , Kernel Time: 17179869184.000000, User Time: 21475323904.000000, Wait Time: 0.000000

- Registers -
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000000000000 rsp=0000000000000000 rbp=0000000000000000
cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000

- Callstack -
ChildEBP RetAddr  Args to Child
(-nosymbols- PC == 0)
00000000 00000000 00000000 00000000 00000000 00000000 !+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
ID: 95971 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1673
Credit: 17,596,129
RAC: 22,414
Message 95972 - Posted: 4 May 2020, 1:34:19 UTC - in response to Message 95971.  
Last modified: 4 May 2020, 1:48:26 UTC

Oh dear, same errors on my main rig... All three of them
Well, I'll stop touching the target runtime setting when there are tasks running, then.
Possibly a side effect of the way the new Estimated completion time fix using the Target CPU time has been implemented/interacts with the existing Runtime/ Watchdog timer limit .

Changing the Target CPU runtime needs to change the wu.fpops_bound value, which is based on the wu.fpops_est (Estimated computation size) which for Rosetta must have been customised to work with Tasks that have set Runtimes.
Going from a 8 hour Target CPU runtime to a longer Target CPU runtime needs to also increase the wu.fpops_bound bound value so the Task doesn't error out as yours have before the Watchdog timer is able to kick in,

Something like
Maximum runtime limit = (Estimated completion time value + Watchdog timer value)*15%

So the for the default 8 hour Target CPU runtime Task.
(8+10)*15% = 20hrs 42min before it errors out.

36 hr Target CPU time
{36+10)*15% = 52hrs 54min before it errors out.

2hr Target CPU time
(2+10)*15% = 13hrs 48min before it errors out.
Grant
Darwin NT
ID: 95972 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95973 - Posted: 4 May 2020, 2:17:22 UTC

Sorry, that limit should be increased on Ralph@h for new jobs now. FYI, I updated both the scheduler and the validator on R@h. So hopefully these updates will address the job cache and the crediting issues that Tomcat has highlighted.
ID: 95973 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Number crunching : Possible bug in the "Average Processing Rate" calculation.



©2024 University of Washington
https://www.bakerlab.org