ReactOS Newsletter: Newsletter 74

Author: Z98. Link to original: (Russian).
Tags: FOSS, GPL, Newsletter, ReactOS, windows Submitted by evilslon 09.07.2010. Public material.
Newsletter 74 by Z98 on 2010-07-09 - General Developments - Yarotows - Memory Corruptions - Filesystem Notifications

Translations of this material:

into Russian: Выпуск новостей ReactOS №74. Translation complete.
Submitted for translation by evilslon 09.07.2010 Published 6 years, 11 months ago.


General Developments

Amine Khaldi's effort in getting the driver headers cleaned up is starting to show results. Not only are a good portion of the driver headers sorted out, the necessary checks have been added to allow the headers to be used in both MSVC and GCC. Anyone who has had to write cross platform code will know that the two compilers have subtle differences at times in their allowed syntax so producing code that works with both can get nontrivial quickly. Amine's work has already helped with the integration of some new drivers as well as getting the ARM kernel to compile further.

The ReactOS web team is looking into the issues with both the buildbot and the site's DNS. So far the DNS issues seem to have been resolved and the site is once again accessible through its domain name. The build server on the other hand suffered a major failure with its processors and will likely take a while before it is back up. The testers have cobbled together a temporary system for their purposes so regression tests are still running. The server failure also presented Olaf Siejka, one of the primary testers, a chance to try out a replacement notification system for builds and general assistance in the ReactOS IRC channel.


Yet Another Rewrite of the Old Win32 Subsystem is a very appropriate name for the branch Jérôme Gardou and Timo Kreuzer have been working on for the past several months. What started as a rewrite of how display drivers were loaded by the Win32 subsystem by Timo ended up being a fairly substantial project to fix major issues in the subsystem implementation. These issues were directly responsible for ReactOS' inability to do something as mundane as changing the screen resolution. In many ways Yarotows is the response of a few developers skeptical of the ARWINSS approach, but the difficulty of fixing win32k's issues in trunk without causing massive breakage resulted in some reluctance to have it undergo major surgery. With a branch however, the developers are free to break whatever they need to finally fix some of ReactOS' longstanding architectural issues.

In Windows, the GRAPHICS_DEVICE data structure is used to represent each of the hardware adapters present. The adapters themselves may support a variety of modes that may draw things using different techniques, which are exposed through display drivers listed in the GRAPHICS_DEVICE structure. The display drivers are themselves represented by the logical device object (LDEVOBJ), which lists all the modes supported by that particular driver. There is also a physical device object (PDEVOBJ) representing the current mode being used and provides the surface to which the display driver draws to. When the Win32 subsystem starts up it looks through the registry to see which devices are present and creates a GRAPHICS_DEVICE object for each one. Then a mode is selected and the appropriate LDEVOBJ and PDEVOBJ loaded. The Win32 subsystem in trunk however did not have LDEVOBJ implemented, which pretty much made its loading mechanism completely wrong architecturally and also rendered it unable to switch between different display modes without rebooting since available modes are supposed to be kept track of by the LDEVOBJ. The entire loading system basically had to be implemented, a highly nontrivial task and resulted in major breakages while it was undertaken. However, with the various objects implemented mode changes are one step closer.

Implementing the correct driver loading architecture ended up revealing several bugs that were never exposed since mode switching never worked in the first place. Fortunately MSDN provided a detailed description for how the mechanism is supposed to work and served as a useful reference for filling in the holes. The first dealt with access to the surface for drawing. Access had to be controlled as allowing two different things to draw to the surface at the same time could cause problems. Changing the mode while someone is drawing to the surface could also result in very bad things happening, as the surface's properties may not be the same anymore. Previously two locks were used, one protecting the surface and the other the device context. While this protected against concurrent drawing to the surface, the locks would fail as soon as the mode was changed. The lock on the surface was changed to one on the PDEVOBJ, which when acquired during a mode change will prevent anyone from using the PDEVOBJ or its associated surface. That this was never encountered or at least never caused visible problems is likely due to the fact that no one ever got far enough in a mode change to trigger any deadlocks, or even if they did something else would have render the system unusable. The lock on the surface has been removed, leaving the lock on the PDEVOBJ to guard against simultaneous drawing attempts.

Another issue that cropped up had to do with palettes. Palettes are used to help interpret the bits on a surface, specifically what color the values represent. Their implementation in ReactOS is also spotty, which meant the features needed to support mode changing was also missing. The biggest problem occurred when the mode change also changed the bit representation of depth, which would invalidate the palette that the PDEVOBJ was holding onto. Unfortunately surfaces were still relying on that palette and would end up with completely wrong colors. To get around this the palette was tied to the surface, which would change with the mode change and have the right information.

Besides the fixes to provide support for mode changing, several other long standing issues have also been dealt with. Under very specific conditions, Win32k would find itself in a situation where multiple threads would attempt to acquire exclusive locks on multiple objects in different orders. This condition would have immediately led to a deadlock as neither would release the locks they already held. As unlikely as the conditions were, locking should never depend on such uncertainty as the entire system would fall apart if the conditions were ever met. The fix applied here was to follow a very specific order for acquiring locks, designed to avoid entering into such situations in the first place. The other change made in the branch is a performance enhancement. Bitmaps were previously allocated from the paged pool, which despite its name is still a scarce resource and should not be wasted. In the branch bitmap allocations were changed to the general system memory. Windows actually allocates them from session specific memory, but ReactOS does not have enough of session support implemented yet to do this. Still, it does remove one more contender for pool memory.

Memory Corruptions

Not too long ago Aleksey Bragin implemented a pool integrity checker to try and find data corruption in ReactOS. This checker worked by padding pool allocations and checked if any write was made to the extra pages. However, Aleksey's checker will no longer work due to major changes made to the pool manager. Michael Martin now has a system for at least checking nonpaged pool. Michael's technique involved taking advantage of a feature in QEMU that lets GDB connect to it and watch memory access. Michael then hardcoded his custom build of QEMU to watch a memory location in ReactOS that pointed to the address of the block of memory allocated from the pool. Every time the nonpaged pool was touched by ReactOS, Michael had QEMU walk a list of stored addresses to make sure the operation was happening to an actual block. Needless to say this severely slowed the system and it took ReactOS about five minutes to boot to the GUI. When an improper memory access occured, Michael had QEMU crash the OS and dump the user into the debugging interface, allowing him to examine the source of the access.

So far Michael has come across four bugs caused by pool corruptions, three of which are from his own code. The first was some leftovers from his fix to the nonqueued message pump mentioned in the newsletter 72, where bookkeeping information was still being written about messages that did not go through the queued message pump. The other two problems were in Michael's new USB code, one being a simple typo and another an offset calculation mistake. All three have been fixed. The last issue still being investigated is UniATA causing a pool corruption. That is still being investigated but hopefully with his new tool Michael will also resolve it soon, albeit at a very slow pace.

Another area suffering from memory corruption issues is trying to play sound on real hardware. The testers were trying to get drivers working but kept running into crashes as somewhere in the kernel memory is being corrupted. The recent work on the nonpaged pool by the ARM team seems to be what is exposing the corruption but so far Johannes Anderwald has not been able to find their cause. Unfortunately Michael's tool will not be able to help testing this directly as it is QEMU specific. However, every corruption he does find may be the one the testers are encountering.

Filesystem Notifications

Though Pierre Schweitzer has been fairly quite for some time now, he has been busy studying and implementing filesystem notifications for ReactOS. The lack of notifications are directly responsible for several issues in ReactOS, especially in the shell. One of the clearest examples is when a program is being uninstalled, one should be able to see the program's directory being deleted. This is due to the filesystem informing the shell it is deleting a directory, but without notifications this is not possible. Other issues would be apparent except other bugs and missing functionality preempt them.There are two types of notifications, one that goes through the Plug and Play path to inform the system of changes to the state of a disk volume, such as mounting and dismounting of a USB drive. The other issues notifications about changes to the contents of a directory.

Of the two, Pierre has managed to partially implement state changes. His effort was significantly hampered by the crude state the ReactOS PnP system is in. This is due to all notifications that are issued to inform interested programs and users of state changes are routed through the PnP manager. When a program informs the OS it wants to be aware of state changes, it is the PnP manager that receives this registration and maintains a list of all interested parties.

File notifications still needs some work, however Pierre has at least figured out the fundamentals. A while back mention was made of the FileSystem Runtime Library, also a project of Pierre's. One of FsRtl's purposes is to serve as an entry point for notifications, both file and state. The actual dispatching of notifications depends on the type, with the PnP manager dealing with state notifications as mentioned above and the FsRtl itself handling file notifications. When programs request to be notified of changes, the filesystem driver is sent an I/O Request Packet to signify what the program wants to be notified about. The driver will then register that notification in a list maintained by the FsRtl which is used to search for interested parties whenever files are modified. If a match is found, the FsRtl fills in the necessary details and completes the IRP. Pierre has at least gotten notification registration implemented and sitting in a patch and is still working out the actual sending of notifications. Once finished, Pierre can move onto the other filesystem issues ReactOS has.