Looks like there are a couple issues here:
1) Your GEM PATHS are not in your SHELL PATH
In a terminal window, if you run
echo $PATH you need to see one (or more) of those GEM PATHS listed in the PATH output to run middleman.
2) System Ruby versus RVM
When you run
which ruby, you're seeing the default Ruby version OS X ships with, but it's not typical for Ruby developers to use that Ruby. RVM or rbenv allow you to install and manage a bunch of Ruby releases separately from system, which is a good practice to follow.
You have RVM installed, so you should set your default RVM ruby and use RVM instead of system:
Once you can run
which ruby and see something like this below, things will be better:
After you are using an RVM ruby, re-install middleman:
gem install middleman