If we step back for a moment and think about what a world with "classic" wizards look like they are basically academics. If anyone could potentially learn wizardry then the world would be very different. Eventually magic just becomes/replaces technology. Uhg. Boring. Robots and trains and communicators and just fucking shoot me. Magic missile was kind of cool until everyone had it. In 5e low-level wizards (et. al.) are walking guns.
Sure, you can wrap role playing and setting and everything around it, but it still never comes across as dangerous. Wild magic came into the fold. Interesting. The Deck of Many Things and the Wand of Wonder? Cool - but a smart player would just burn that shit and walk away.
I really dug on Ars Magica's idea of crafting spells that suit your needs. But again - danger. If there is just a chance of shit going wrong that is also just purely random chance. DCC did a pretty good job of making things much more interesting, but it is still Vancian magic and wizards pretty much dominate the game.
If you like those things, cool. Not me. Ages ago I added in "free casting" with the chance of things going wrong. It was a massively clunky add-on to D&D/Hackmaster. Eventually I wrote the system we are using in Sorrow in Haven. Check it.
- The basics: an opposed roll (like everyhting else): 2d6+arcane vs 2d6+difficulty.
- The spell is built by the player taking looking at a chart and figuring out range, target, duration, and effect. Effects outside of direct numbers (like 3d of damage) are a bit subjective and must be related to the wizard's focus (more on that below)
- Total everything up from the chart and that is the difficulty.
- Chances are, a character's arcane attribute will be in the 3-6 range. Difficulty for "average" spells float around 10, but have no ceiling if the player keeps piling on effects
The character has a few "foci". These are effectively words or phrases they must base the spells they cast. For example, in a recent set of games, a wizard with "Barrier of Steel" used it to cast up a steel wall and is working on casting a spell to make their skin steel. Both are super valid.
Now the danger/risk part comes in when you look at the possible results
Action Roll (that 2d6 opposed biz mentioned above) is a Success, the spell goes off. There are benefits for rolling up a Critical or Legendary Success, but those aren't interesting for this. If the spell Fails, there is a chance some Magical Chaos shows up. On a Critical Fail Magical Chaos definitely happens. on a Legendary Failure ALL THE MAGICAL CHAOS happens. At once.
Magical Chaos is represented by a number of cards. One card is presented per focus the wizard has (and some rules for multiple wizards and the like). The players can see the names of the magical chaos, but not what they do. If Magical Chaos happens, they choose a card and deal with the consequences.
The player can spend some of their character's Endurance (stamina)to improve their chances (as with all Action Rolls). They can also spend their Vitality (causing themselves actual wounds) to improve chances of success. This is where the choice and risk come into play.
If they spend too much END and VIT to make sure the spell definitely absolutely goes off, they have weakened themselves for future encounters (think of it as spending you Hit Points). If they don't spend enough there is not just a chance of failure, but a chance of failure with something going WRONG.
So what the hell is Magical Chaos? The players have encountered this twice so far.
One time the wizards tongue swelled up into a huge worm while their mouth disappeared. They had to choose to bite off their own tongue or suffocate. In the end some clever dagger-work did the trick and neither resulted. Role playing to the rescue!
Another time Tiger Mott (a famous wizard of GDD Chapter 938 - the Extravagant Philosophers) cast a spell and all his skin sloughed off. All of it. Beneath the skin he had transformed into a porcelain mannequin. He ended up being even stranger than before - knowing that a single critical attack will shatter his fragile body. Role playing to the ... that was MY character. That sucked, but Tiger sure is interesting now :)
Magic is dangerous. People fear it because if it messes up, monsters can come through and destroy the world.
Magic is powerful. A wizard can attempt to cast anything they can think of, but if they over-reach things go wrong (see Magic is dangerous).
Magic is magical. It is unpredictable and creative instead of a list of things. It has strange effects and often unknown consequences and, at least for non-wizards, it is mysterious and weird.
How does this work in play? Really well! It takes a new wizard player a session or two to wrap their heads around it, but it works and encourages creativity.