Creating mods is done with map editors and compilers, but also involves .... texture creation, making pak files, modeling, skinning, editing sound and graphics files, and writing your own progs.dat with Quake C. Linux support is available, but is not comprehensive.
Quake C is Quake's own game language (it was the first game such), and it allows mods to work seamlessly on all operating systems.
The main map editor for Quake. It is based on id Software's own editor, and requires OpenGL.
A multi-game world editor. It has a stiff learning curve, but is very efficient after you master it.
Linux port of the most used map compilers.
Tyrann's popular map compilers, including qbsp, vis and tyrlite. Supports rotating entities and coloured lights.
LordHavoc's compiler. It can compile leaky or incomplete maps (use only for testing!) and does a ton of other stuff, including coloured lighting.
Command line tools to handle pak and wad files (source code only).
Comprehensive, advanced QuakeC compiler with gui and command line versions. If this one can't compile it, nothing will!
QuakeC compiler
Wine is a project that allows many Windows programs to work with Linux.
Multipurpose texture editing tool. Can batch convert BMP files to Quake textures, etc. Also provides several paint functions, and macros such as rivets and decals.
QuArK is a multipurpose Quake editing tool. It loads, edits and exports multiple file formats, and has matured into a multi-game, multi-format tool. Some functions are broken though, and it may not work with all versions of Wine.
Light tool that does radiosity-based lighting, like Quake II.