Just woke up another machine with a stuck unit. It was stuck at 64K, but after telling it to retry, the new stuck condition is 0.19/496.52 KB for an re12dslf... project.

Some kind of input buffer size problem? Maybe the rosetta@home people are trying to increase the buffer sizes for incoming data? The problem is somehow related to certain work units requesting smaller buffers than they actually need, and then getting stuck because they can't send the rest of their data?
Retried a few minutes later, and the new stuck status is:

0.06/1.62 MB
64.00/511.73 KB
64.00/219.68 KB
64.00/407.93 KB

I am not sure how you are measuring that, but BoincTasks shows that my Ubuntu WU is stuck at 2.039% of 1569.45k (also for an rb ...), which comes out to 32.00k. Since that is half of 64k, it must mean something.
I trust the appropriate expert will figure out what.

I have one too:
wu

SO WHAT TO DO?

---

Hi all you enthusiastic crunchers.....

SO WHAT TO DO?

Option 1: Stay on Rosetta and hope that they fix it.

Option 2: Go to another project and hope that they fix it.

Option 3: ??? (Fill in the blank)

EDIT: I am doing a little of each, keeping a couple of cores on Rosetta but adding TN-Grid to the mix.

---

9 stuck uploads. BOINC no longer receiving tasks from Rosetta.

Have one WU stuck on upload since at least Friday, Other Rosetta tasks seem to complete and upload fine.
If this problem is so elusive, why aren't there any admins/programmers actively communicating with folks that have this problem in order to solve it?

Ralf

Have one WU stuck on upload since at least Friday, Other Rosetta tasks seem to complete and upload fine.
If this problem is so elusive, why aren't there any admins/programmers actively communicating with folks that have this problem in order to solve it?

Ralf

We can see the errors in our logs. I don't know what additional info would help from users but if you have info that may help, please continue to post it here.

The issue manifested at around 11:30 am on April 12. See Luki's (our systems engineer) notes:

1) The problem started on Wednesday 4/12 at 11:30 am PST. All web servers started misbehaving at the same time. Upload timeouts went from 100x (from ~80/day to ~8000/day).
2) Out of ~37000 unique client IPs that uploaded results this week, only ~1400 are affected (4%). So it's not random.
3) The network or machine loads have not really changed.
4) The size of the uploaded results is only ~600 KB, yet the upload stalls after ~8 KB (client stops sending data to server). Hence the upload_handler waits for more data until apache times out the request. The upload handler uses no CPU and causes no IO load (yet).
5) As you know, the web server nodes are directly connected to UW switches; our switched can't be to blame here. Still, I tried moving one of the public IPs (bsrv5) to another server, connected to another switch UW -- the problem moved with it instantaneously.
6) The culprit really seems to be at the network level, like the TCP ACKs don't make it to the client; yet we capture them on the wire. Is UW dropping them? Like they trigger an IDS?

Luki


We are still trying to figure out what is causing this and will keep you all posted if we make progress.

6) The culprit really seems to be at the network level, like the TCP ACKs don't make it to the client; yet we capture them on the wire. Is UW dropping them? Like they trigger an IDS?

Luki


We are still trying to figure out what is causing this and will keep you all posted if we make progress.

I don't really want to volunteer and I don't have any sniffers set up right now (or so I claim), but that seems like something you can test fairly easily from any client on the outside. The way you describe it sounds like a filter is picking up certain ACKs and blocking them, so it seems you should be looking at the characteristics of the ACKs that should be going to the blocked packets in comparison to the ACKs that are making it. The bug seems to be in a can, which is the easy kind to pin down. (At least I have not yet noticed any stuck results getting unstuck, and the late result is apparently going to ignore its deadline.)

Your [Luki's] packet size estimates don't seem to make sense to me, but all of my data definitely disagrees with the 8 K thing... I'm seeing 64 K (about 40 TCP/IP packets?) or variations around 3 times that, and someone else has reported a 32 K failure.

In my own case, I estimate the blocked packets are only around 5%. Also I can say that enough uploads have succeeded from the Mac that I'm pretty sure it isn't affected, which may be another clue.

By the way, I'm just taking the transfer progress from the "Transfers" tab in the BOINC client. Right now this machine shows 4 stuck results, with one showing 0.06 and the other three 64.00.

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)

A little update.

One stuck task was miraculously uploaded yesterday.

16-Apr-2017 17:15:48 [rosetta@home] Started upload of 14dslfv5_14re4np_gb_0037_0001_30_0002_SAVE_ALL_OUT_480050_322_0_0
16-Apr-2017 17:16:02 [rosetta@home] Finished upload of 14dslfv5_14re4np_gb_0037_0001_30_0002_SAVE_ALL_OUT_480050_322_0_0
16-Apr-2017 17:16:06 [rosetta@home] Sending scheduler request: To report completed tasks.
16-Apr-2017 17:16:06 [rosetta@home] Reporting 1 completed tasks
16-Apr-2017 17:16:06 [rosetta@home] Not requesting tasks: don't need
16-Apr-2017 17:16:09 [rosetta@home] Scheduler request completed

https://boinc.bakerlab.org/rosetta/result.php?resultid=910051019


Now I have 4 remaining tasks stuck on uploading state.
You can see that they break the "8 KB rule".
https://s28.postimg.org/43l918fy5/rosetta_stuck_tasks.png
Maybe 3 tasks stalled 4 times, so that's why uploaded file size is 32KB (=4*8KB) and not 8KB.

Luki was referring to a specific job and was not speaking generally. The data size is variable.

I think this issue is not just affecting uploads, when I tried my last post on this thread last night it timed out and the data was truncated. I had to manually remove the truncated post and repost.

Luki was referring to a specific job and was not speaking generally. The data size is variable.

I think this issue is not just affecting uploads, when I tried my last post on this thread last night it timed out and the data was truncated. I had to manually remove the truncated post and repost.

Sorry about my confusion, but I keep saying I don't want to spend much time on this as I keep spending time on it... I should have reread Luki's reply more carefully. However, I'm pretty sure that I have seen some stuck packets change their sticking points. That was a few days ago, but right now on the two machines at hand I see 6 stuck results, and all of them are at 64.00 or 0.06 (as reported for the larger results).

If the lost ACK packets can be identified, perhaps they can be padded or unpadded differently? Something to change whatever characteristic the unknown filter is blocking them for? Ugly suggestion in the neighborhood of network internals, but sometimes that's where the spamming scammers (or worse) force us to go...

By the way, I'm still not convinced it isn't Windows-10-specific. Yes, there was at least one report of a similar problem on Linux, but maybe it was different. I ran one of my Linux boxes all day yesterday without getting a sticker, and my Mac remains sticker free. Firing up a different Linux box today and will let it run all day (but that's including a major upgrade, which confuses all issues).

One more thought... More generalized network problems affecting other connections? Naw, I don't think I want to go there. At least not yet.

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)

By the way, I'm still not convinced it isn't Windows-10-specific. Yes, there was at least one report of a similar problem on Linux, but maybe it was different. I ran one of my Linux boxes all day yesterday without getting a sticker, and my Mac remains sticker free. Firing up a different Linux box today and will let it run all day (but that's including a major upgrade, which confuses all issues).

Today I updated my Ubuntu machine from 16.10 to 17.04 in the hope that it would unstick it, but it did not. However, three other Rosetta WUs have finished and uploaded on that PC, so it is not getting worse, and I will just let it run. Since it is quite unusual for Rosetta, and started at the same time as on my Win7 machine (no longer on Rosetta), it would appear to be the same problem to me.

---

By the way, I'm still not convinced it isn't Windows-10-specific. Yes, there was at least one report of a similar problem on Linux, but maybe it was different. I ran one of my Linux boxes all day yesterday without getting a sticker, and my Mac remains sticker free. Firing up a different Linux box today and will let it run all day (but that's including a major upgrade, which confuses all issues).

My host (4 stuck task now) has Xubuntu 14.04.5, Linux kernel 3.13.0-116-generic.

I'm getting 1 WU that hangs for every 40-50 that process normally (under Linux).

Just a thought - have you guys taken a WU known to have the problem, and tried running it on another host to see if anything looks unusual?

As mentioned before, I doubt that this is something networking/IP related or protocol/IDS related, as WUs from the same host get uploaded just fine.

And it is certainly not Windows 10 related, as someone else mentioned in his response, as in my case, the host in question is Windows 7 Pro/64 bit...

And those 3 WUs on the other laptop that I mentioned now show up with a validation error. strangely enough...

Ralf

I have two WUs that have been stuck uploading (one @ ~34% and the other @ 100%) for several days now. Other WUs have been uploaded and credit received with no problem.

I'm running a Mac, btw - Sierra v10.12.4

Keith

Your [Luki's] packet size estimates don't seem to make sense to me, but all of my data definitely disagrees with the 8 K thing... I'm seeing 64 K (about 40 TCP/IP packets?) or variations around 3 times that, and someone else has reported a 32 K failure.

Yes, all mine are a consistent and small size. 32k for me for what that information's worth. And all on Windows 7 machines.

Lots of tasks still going through fine. But when one task has a problem it keeps on with that problem for days on end - hence my idea of a flag getting set somewhere. I upload 24 tasksday. Total of 3 problems, so about 45 go through straight away.

---

Well, I'm pretty well convinced it is affecting all rosetta@home projects on all of the major platforms. Apparently I'm just lucky that I haven't seen one on my Mac or Linux boxen, but most of my machines are running Windows 10 most of the time. I have at least seven of the stuck-on-uploading results now.

Still not convinced that it isn't something at the network level, however.

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)

Another (stuck) task got sent this night.

18-Apr-2017 04:05:28 [rosetta@home] Started upload of rb_03_23_72525_116778__t000__ab_robetta_IGNORE_THE_REST_474917_815_0_0
18-Apr-2017 04:05:46 [rosetta@home] Finished upload of rb_03_23_72525_116778__t000__ab_robetta_IGNORE_THE_REST_474917_815_0_0
18-Apr-2017 04:05:48 [rosetta@home] Sending scheduler request: To report completed tasks.
18-Apr-2017 04:05:48 [rosetta@home] Reporting 1 completed tasks
18-Apr-2017 04:05:48 [rosetta@home] Not requesting tasks: don't need
18-Apr-2017 04:05:51 [rosetta@home] Scheduler request completed

https://boinc.bakerlab.org/rosetta/result.php?resultid=910050184


Why are some tasks uploaded successfully after days?


Now I have 3 remaining (stuck) tasks.

Di 18 Apr 2017 17:28:24 CEST | rosetta@home | Started upload of 12dslfv7_gb_0003_0005_11_0001_SAVE_ALL_OUT_480505_41_0_0

Doesn't work. 64,00 / 809,82 KB, then stops...

Looks like it's been fixed, but I'm still curious what the problem was. My stuck-on-uploading results started clearing yesterday, and the first four machines I've checked are all good to go now.

Anyone still having problems?

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)