Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Hang in DviDevice::Destroy
19-07-2012, 02:19 PM
Post: #1
Hang in DviDevice::Destroy
I've had reports from multiple Mac OS X users of a hang while destroying the MediaServer device. I'm not seeing this problem myself, which makes debugging a bit difficult. From looking at a thread dump created by one of the users with this problem, I've established that it's hanging in DviDevice::Destroy on the call to iShutdownSem.Wait().

The device was enabled before my Java code called device.destroy(). I've looked at the code and stepped through it in the debugger, and I couldn't see any obvious problems in the code. I'm running on a code level from the end of April, so I've also looked at the most recent DviDevice.cpp code on Github. The only relevant change seems to be the addition on 5th July of this code:
Code:
// Tell services not to accept further action invocations
for (TUint i=0; i<iServices.size(); i++) {
    iServices[i]->Disable();
}

Could the lack of the above code possibly allow an invocation to re-enable the device during the destruction sequence, causing iShutdownSem to be cleared? It seems unlikely to me that multiple people would see this as a consistent problem, but I don't have any other ideas.

Any suggestions or pointers would be much appreciated.
Find all posts by this user
19-07-2012, 03:24 PM (This post was last modified: 19-07-2012 04:28 PM by simonc.)
Post: #2
RE: Hang in DviDevice::Destroy
I don't think the 5th July change will be relevant here. It ensures that DviDevice::Disable blocks until any in-progress actions complete. This fixes some possible bugs but potentially introducing more delays isn't going to help with your hang...

You mention a thread dump. Is there any chance you could mail/PM a copy? I'd guess the hang is a deadlock, probably around the time the last byebye message is sent. The thread dump would be very useful in confirming or disproving this.

If we don't find any other solution and if the hang only happens on shutdown of minim server, you could hide the problem by skipping ohNet destruction and rely on the OS to clear up all resources when the process exits. This approach is a bit unpalatable (it risks hiding resource leaks in your app and may introduce network address in use problems for rapid re-launches) but might do in the worst case.
Find all posts by this user
19-07-2012, 05:46 PM
Post: #3
RE: Hang in DviDevice::Destroy
(19-07-2012 03:24 PM)simonc Wrote:  I don't think the 5th July change will be relevant here. It ensures that DviDevice::Disable blocks until any in-progress actions complete. This fixes some possible bugs but potentially introducing more delays isn't going to help with your hang...

You mention a thread dump. Is there any chance you could mail/PM a copy? I'd guess the hang is a deadlock, probably around the time the last byebye message is sent. The thread dump would be very useful in confirming or disproving this.

If we don't find any other solution and if the hang only happens on shutdown of minim server, you could hide the problem by skipping ohNet destruction and rely on the OS to clear up all resources when the process exits. This approach is a bit unpalatable (it risks hiding resource leaks in your app and may introduce network address in use problems for rapid re-launches) but might do in the worst case.

I'll email you the thread dump. I looked for a deadlock myself, but I didn't see one. I suppose if something went wrong with sending out the byebye message(s) and DviDevice::ProtocolDisabled didn't get called, this would cause the observed hang.

It isn't possible to skip the destroy() call because it happens when stopping and restarting the music server component. MinimServer is built using a multi-component architecture, and some of the components are designed to be stopped and restarted without ending the whole application.
Find all posts by this user
20-07-2012, 12:35 PM
Post: #4
RE: Hang in DviDevice::Destroy
(19-07-2012 05:46 PM)simoncn Wrote:  I'll email you the thread dump. I looked for a deadlock myself, but I didn't see one. I suppose if something went wrong with sending out the byebye message(s) and DviDevice::ProtocolDisabled didn't get called, this would cause the observed hang.

Thanks. You're right that the thread dump doesn't show any obvious problem. It is slightly unusual however in that it shows 3 multicast listeners running, which implies the host device has 3 active subnets.

I've spotted one possible problem in the code. If the system clock moves backwards while byebye messages are being sent, the remaining messages will be spread out over the amount of time we've lost. I'll add a fix for this but it seems pretty unlikely this'd be the cause (its unlikely to happen once during the 50ms duration of byebyes; repeated instances seem improbable).

If the hangs recur on any systems, it might be interesting to ask that user to try disabling all but one subnet to see if that helps.
Find all posts by this user
20-07-2012, 04:43 PM
Post: #5
RE: Hang in DviDevice::Destroy
(20-07-2012 12:35 PM)simonc Wrote:  Thanks. You're right that the thread dump doesn't show any obvious problem. It is slightly unusual however in that it shows 3 multicast listeners running, which implies the host device has 3 active subnets.

I've spotted one possible problem in the code. If the system clock moves backwards while byebye messages are being sent, the remaining messages will be spread out over the amount of time we've lost. I'll add a fix for this but it seems pretty unlikely this'd be the cause (its unlikely to happen once during the 50ms duration of byebyes; repeated instances seem improbable).

It seems unlikely that the system clock would move backwards. I suppose this could happen if the user manually resets it to an earlier time, or if the computer resyncs with a network time server which is set to an earlier time. The odds of this happening every time the user restarts MinimServer seem astronomical. (Some users get this hang on 100% of these restarts.)

Quote:If the hangs recur on any systems, it might be interesting to ask that user to try disabling all but one subnet to see if that helps.

I'll make this suggestion to the user who provided this thread dump, and I'll let you know the results.
Find all posts by this user
21-07-2012, 07:57 AM
Post: #6
RE: Hang in DviDevice::Destroy
(20-07-2012 04:43 PM)simoncn Wrote:  
(20-07-2012 12:35 PM)simonc Wrote:  If the hangs recur on any systems, it might be interesting to ask that user to try disabling all but one subnet to see if that helps.

I'll make this suggestion to the user who provided this thread dump, and I'll let you know the results.

Here's the reply:

I am no network specialist, but I have indeed three active network adapters, although on the same subnet as far as I can see. Two of them seem to have been added as part of my Parallels installation. After I deactivated the two Parallels network adapters, I was able to stop, restart and exit minimserver.

Would you be able to install Parallels and try to reproduce and debug this problem?

As a further data point, I've tried enabling two active adapters (Wi-Fi and Ethernet) on my Macbook Pro, but without Parallels. I've tried this with the adapters on the same subnet and with the adapters on different subnets. This doesn't reproduce the problem.
Find all posts by this user
21-07-2012, 08:25 AM
Post: #7
RE: Hang in DviDevice::Destroy
The Cling UPnp framework specifically discards Paralllels virtual interface as not suitable for UPnP. It also discards VMWare interfaces.

See this file, function isUsableNetworkInterface(), on how to detect such interfaces (based on interface name):

https://github.com/bubbleguuum/cling/blo...yImpl.java
Find all posts by this user
21-07-2012, 08:28 AM
Post: #8
RE: Hang in DviDevice::Destroy
Is there any chance your users shut minim server at the same time as another operation which changes the number of available subnets? (Devices operating on 3 subnets presumably have some VMs running which operate virtual network adapters so system shutdown at least would generate subnet change events.)

I've spotted another problem where a subnet change event while byebye messages are being sent would prevent a DviDevice being notified that an async disable had completed.
Find all posts by this user
21-07-2012, 09:24 AM
Post: #9
RE: Hang in DviDevice::Destroy
(21-07-2012 08:28 AM)simonc Wrote:  Is there any chance your users shut minim server at the same time as another operation which changes the number of available subnets? (Devices operating on 3 subnets presumably have some VMs running which operate virtual network adapters so system shutdown at least would generate subnet change events.)

I've spotted another problem where a subnet change event while byebye messages are being sent would prevent a DviDevice being notified that an async disable had completed.

I'm almost sure this isn't happening. From the user reports I've received, it's my understanding that the user is restarting/recycling the music server component of MinimServer (by selecting Restart from the MinimServer icon) and isn't doing any other operations at the same time. There's definitely no system shutdown or restart involved. I'll request positive confirmation that no other operations are happening while MinimServer is being restarted.
Find all posts by this user
22-07-2012, 07:33 PM (This post was last modified: 22-07-2012 07:35 PM by simoncn.)
Post: #10
RE: Hang in DviDevice::Destroy
I've had confirmation that nothing was happening on the Parallels adapters at the time of the destroy() call.

Another Mac user has reported the same problem with additional adapters created by VMware Fusion.
Find all posts by this user


Forum Jump: