DataMapper 0.10+ Basic Sphinx Support
I needed sphinx support and I needed it now. Apparently dm-sphinx-adapter wasn't ported to dm 0.10. I started porting it, but I kind of felt like I was going down a rabbit whole. I will continue to work on it over the next couple days, in the meantime at Vokle we used this to get our fulltext on. Note, this is doing the fulltext search across all columns in your fulltext index. Its not doing cool matchers like :updated_at.gt yada-yada. It also doesn't implement a DataMapper Adapter, its a super hack, but it gets the job done for us.
Give me your two cents and stay tuned for a full blown adapter.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | module DataMapper module Model def sphinx_setup(opts={}) @_sphinx_index = opts[:index] end def search(_query_, opts={}) sph_client = opts.delete(:client) || Riddle::Client.new(*SimpleSphinx.conf) sph_client.match_mode = opts.delete(:match_mode) || :extended sphinx_query = _query_.to_s.gsub(/[\(\)\|\-!@~"&\/]/){|char| "\\#{char}"} results = sph_client.query(sphinx_query, @_sphinx_index || self.storage_names[:default].to_sym) matches = results[:matches] ids = matches.map{|match| match[:doc]} # Fetch self.all(opts.merge({:id => ids})) end end end class SimpleSphinx class << self def conf [server, port] end def server=(s) @_server = s end def port=(p) @_port = p end def server @_server || 'localhost' end def port @_port || 3312 end end end |
Wanna use it?
1 2 3 4 5 6 7 8 9 10 | # Your Model here Person.sphinx_setup(:index => "indexName") #defaults to storage name SimpleSphinx.server = "example.com" SimpleSphinx.port = 3312 Person.search("cory") #=> DataMapper::Collection # you can pass an additional hash of stuff you would pass to Model#all and this will be used to filter # the result set down POST sphinx when its collecting data from datamapper Person.search("cory", :limit => 10, :gender => :male) |
You've got your sphinxter back. Congrats!
DataMapper and Merb, sharing your errors via the merb display API.
This is a quick little snippet. At Vokle all of our products are built using our core API, which thanks to merb, was a piece of cake. The thing that sucks is sometimes we return objects as JSON via the display API and the "errors" are missing in the event that validation failed. How to fix that?
Throw this somewhere:
1 2 3 4 5 6 7 8 9 10 | module DataMapper module Validate class ValidationErrors def to_json @errors.to_hash.to_json end end end end |
Now in your merb controllers, when you are displaying an object that may have errors:
1 2 3 4 5 6 7 8 | class People < Merb::Controller def create(person) #... *SNIP* ... display @person, nil, {:methods => [:errors]} #... *SNIP* ... end end |
And whatever is getting your data back in XML or JSON (yeah or YAML, right) will get the errors on your object as well. Cool.
Yay, users give you invalid data. Congrats.
DM Cutie, et al. 0.4 release (minor updates)
I released dm-cutie, dm-cutie-ui, and dm-cutie-extras with some minor releases on gemcutter.org today.
The minor updates include a lot of help information and some tweaks to dm-cutie-ui to make it easier to use/understand (I hope).
+ add 'help' area describing all the default views
+ MysqlIndex/ Steps column info in help
+ Sort Tables Repo, Gen, Executed, Others by alpha (routes key should be view_name)
+ menu cleanup
+ @records count on page
+ Moved connected to
+ Changed word on relationship link
+ Format of page title
+ Removed duplicate column on mysql_index
+ Sort buttons moved
If you haven't installed DM Cutie, make sure your rubygems is up to date and do:
1 2 3 4 5 | gem install gemcutter --source http://gemcutter.org gem tumble gem install dm-cutie -v 0.4.0 gem install dm-cutie-ui -v 0.4.0 gem install dm-cutie-extras -v 0.4.0 |
Information on setting DM Cutie up is in this post.
If you want the source its available on my github page.
I'll be adding some more interesting tracking stuff for MySQL in the next few days. Now that Vokle has officially launched, I find myself with more time to work on my projects.
When I started this project I really wanted it to be a cool way to get information on all your DM storage needs. I am now realizing that DM Cutie should probably only support SQL and DO backed stores, and I'll be pairing down the internals over the next few days with that in mind.
Most of my database experience as far as optimizing goes is in MySQL, if anyone has some PostgreSQL or SQlite awesomeness they'd like to share, I'd really appreciate it.
DataMapper Cutie – The query tracker and profiler
So Ive been busting my butt on this query tracker for a while now. Its been finished twice and not released (once when I hated the API, and once right before 0.10 was release). But here she is, open for the hacking.
Its pretty easy to get set up, the gems are on http://gemcutter.org
To get started:
1 2 3 4 5 6 7 8 | $ gem sources -a http://gemcutter.org # Add gem cutter to your gem sources $ gem install dm-cutie # This is a plugin pack for dm-cutie $ gem install dm-cutie-extras # this is the front-end to dm-cutie $ gem install dm-cutie-ui |
If you plan to use MySQL to store dm-cuties information, set up a schema for that now by doing:
1 | mysql: CREATE SCHEMA `dm_cutie`; |
dm-cutie will allow you to store stuff across different adapters, so you can actually profile a Mysql DataMapper Repository into a dm-cutie sqlite3 repository. Its ok, she gets it
Now, in your application after your default DataMapper repository is set up do:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | require 'dm-cutie' #currently only data_objects is supported DataMapper::Cutie.enable_adapter :data_objects require 'dm-cutie-extras' # Query Plans! DataMapper::Cutie::Extras.load :mysql_execution_step # Index optimization tips! DataMapper::Cutie::Extras.load :mysql_index # Mysql Errors and warnings! DataMapper::Cutie::Extras.load :mysql_warning DataMapper::Cutie.setup do |c| c[:repo_name] = :dm_cutie # name of your repo c[:exclude_models] = false #[:person, :car] #models to exclude c[:only_models] = false #[:article, :address] #model to include c[:slow_query_length] = 2 end #Forces Cutie to always migrate HER models DataMapper::Cutie.start(true) |
Let dm-cutie sit in your code for a while, she will start to generate lost of data on your repository. I would recommend not putting dm-cutie in a production environment due to the amount of queries that it generates. It is best used in a development or a staging environment and then applies the learned optimization strategies to your production environment.
After cutie has been running a while (or immediately if you are impatient) from the command line do:
1 | $ dm-cutie-ui -p 4567 --extras=mysql_warning,mysql_index,mysql_execution_step |
If you didn't use any of the extras in your application you SHOULDNT list them when you start the dm-cutie-ui.
That's that. You just need to sign in using the URI for your repository, it will be something like: mysql://root@localhost/dm_cutie.
An important note is that when you log into dm-cutie-ui you are logging into the dm-cutie repository, not your applications, so keep that in mind so you dont get errors saying the dm-cutie repo doesn't exist.
Well that's it for now, I'll post about this more after the holiday when I make a few more plugins. Enjoy!
There is a ton more documentation at:
DataMapper Remixable Updates
DM Remixables 0.9.7 RC 2 is out. Includes clean accessor name creation, more specs, and the ability to assign methods from the Remixable to generated and remixing classes. Woot, woot. See the specs and readme for examples.
DataMapper is Viewable
Even when I'm panicking I'm a fan of datamapper. I like how clean it generally makes my code, but I do notice that I tend to have the same queries cropping up from time to time either in the same application or in other apps using my model library. This just wasn't dry enough for me, so I made dm-is-viewable. It gives sql-like view functionality to DataMapper Resources. It's a pretty simple plugin, but lets take a look, um kay?
Let's start with a really simple User class...
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | class User include DataMapper::Resource belongs_to :location property :id, Serial property :name, String property :username, String property :password, String property :gender, String property :age, Integer property :favorite_color, String property :favorite_number, Integer end class Location include DataMapper::Resource has n, :users property :id, Serial property :name, String property :desc, String end |
Making your resource viewable is pretty easy, first grab the dm-is-viewable gem, then say that your resource is viewable. Views take the exact same parameters that you could pass to Resource.all. The only difference is that dm-is-viewable stores them until they are called.
1 2 3 4 5 6 7 8 9 10 | class User include DataMapper::Resource is :viewable #lets create some views create_view :legal_women, :gender => 'female', :age.gt => 18 create_view :serial_killers, :favorite_number => 666, :favorite_color => 'black' #... Resource code *SNIP SNIP* end |
See the befores and afters below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # Legal Women Before User.all(:gender => 'female', :age.gt => 18) # Legal Women After User.view :legal_women # Serial killers before User.all(:favorite_number => 666, :favorite_color => 'black') # Serial killers after User.view :serial_killers # You can also pass further limiting query parameters to #view. # Legal Local Women Before User.all(:gender => 'female', :age.gt => 18, User.location.name => 'Los Angeles') # Legel Local Women After User.view :legal_women, User.location.name => 'Los Angeles' # Passing additional parameters FURTHERS the limitation of records, so.. User.view :legal_women, :gender => 'male' # => Would return nil, the query would essentially be generated as: # SELECT * from users where gender = 'male' and gender = 'female' |
Simply, clean, useful. Woot.
