There are several vendors, who ship systems with Ubuntu or other distributions of GNU/Linux pre-installed. You might pay more for the privilege, but it does buy a level of peace of mind, since you can be sure that the hardware is well-supported by GNU/Linux.
If you do have to buy a machine with Windows bundled, carefully read the software license that comes with Windows; you may be able to reject the license and obtain a rebate from your vendor. Searching the Internet for “windows refund” may get you some useful information to help with that.
Whether or not you are purchasing a system with Linux bundled, or even a used system, it is still important to check that your hardware is supported by the Linux kernel. Check if your hardware is listed in the references found above. Let your salesperson (if any) know that you're shopping for a Linux system. Support Linux-friendly hardware vendors.
Some hardware manufacturers simply won't tell us how to write drivers for their hardware. Others won't allow us access to the documentation without a non-disclosure agreement that would prevent us from releasing the Linux source code.
Since we haven't been granted access to the documentation on these devices, they simply won't work under Linux. You can help by asking the manufacturers of such hardware to release the documentation. If enough people ask, they will realize that the free software community is an important market.
A disturbing trend is the proliferation of Windows-specific modems and printers. In some cases these are specially designed to be operated by the Microsoft Windows operating system and bear the legend “WinModem” or “Made especially for Windows-based computers”. This is generally done by removing the embedded processors of the hardware and shifting the work they do over to a Windows driver that is run by your computer's main CPU. This strategy makes the hardware less expensive, but the savings are often not passed on to the user and this hardware may even be more expensive than equivalent devices that retain their embedded intelligence.
You should avoid Windows-specific hardware for two reasons. The first is that the manufacturers do not generally make the resources available to write a Linux driver. Generally, the hardware and software interface to the device is proprietary, and documentation is not available without a non-disclosure agreement, if it is available at all. This precludes it being used for free software, since free software writers disclose the source code of their programs. The second reason is that when devices like these have had their embedded processors removed, the operating system must perform the work of the embedded processors, often at real-time priority, and thus the CPU is not available to run your programs while it is driving these devices. Since the typical Windows user does not multi-process as intensively as a Linux user, the manufacturers hope that the Windows user simply won't notice the burden this hardware places on their CPU. However, any multi-processing operating system, even Windows 2000 or XP, suffers from degraded performance when peripheral manufacturers skimp on the embedded processing power of their hardware.
You can help improve this situation by encouraging these manufacturers to release the documentation and other resources necessary for us to program their hardware, but the best strategy is simply to avoid this sort of hardware until it is listed as working in the Linux Hardware Compatibility HOWTO.