If your mod requires FOSE, please provide a link to the main FOSE website instead of packaging it with your mod install. The normal editor can open plugins with these extended scripts, but it cannot recompile them and will give errors if you try. Alternately you can create a shortcut to fose_loader.exe, open the properties window and add '-editor' to the Target field.
Open a command prompt window, navigate to your Fallout 3 install direcory, and type 'fose_loader -editor'. Scripts written with these new commands must be created via the G.E.C.K. If you use a desktop shortcut to launch Fallout 3 normally, just update the shortcut to point to fose_loader.exe instead of Fallout3.exe or FalloutLauncher.exe. Launch Fallout by running fose_loader.exe from the Fallout3 directory. This is usually in your Program Files folder, and should contain files called Fallout3.exe, FalloutLauncher.exe and the G.E.C.K. dll files and fose_loader.exe to your Fallout 3 directory. i copied the install, but you should read it all.