Re: DBIx::Class and caching

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Andreas Pronakis
I was wondering if anyone has tried using the module DBIx::Class::Cursor::Cached but replacing the cache engine Cache::FileCache with Memcache?

I will try it out and let you know the results, but if someone else has already tried it would be nice to know what their findings were

Andreas


--- On Wed, 6/25/08, Oleg Pronin <[hidden email]> wrote:

> From: Oleg Pronin <[hidden email]>
> Subject: Re: [Dbix-class] DBIx::Class and caching
> To: [hidden email], "DBIx::Class user and developer list" <[hidden email]>
> Date: Wednesday, June 25, 2008, 9:23 AM
> These approach caches ready objects, not just a data from
> database, so it
> quite fast, and lets you to cache any data with object,
> even not from DB.


     

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Tobias Kremer
Quoting Andreas Pronakis <[hidden email]>:

> I was wondering if anyone has tried using the module
> DBIx::Class::Cursor::Cached but replacing the cache engine Cache::FileCache
> with Memcache?

Yes, we're using it in production for a site serving approx. 10 million
pageimpressions/month and it never failed on me. I've been told that it's also
in use at Takkle Inc. It works wonderfully for caching stuff that is supposed
to invalidate itself after a certain period of time. It's usage might be a bit
unfamiliar if you want to invalidate stuff manually because you have to have
the exact same ResultSet that was used to fill the cache in the first place.
Other than that I highly recommend its usage because at the moment it seems to
be the only useful working DBIC cache implementation.

I'm setting the cache_object attribute for all resultsets at once via the
schema's default_resultset_attributes() method.

--Tobias

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Oleg Pronin
In reply to this post by Andreas Pronakis
DBIx::Class::Cursor::Cached is too slow for production use. It caches only data from DB making it impossible so store virtual data in objects and after fetching data from cache it has to construct a row object which is a expensive action for CPU, running all the 'inflate_result', and possibly other hard tasks.

When you use DBIC::C::C - it slows down the request by 2 times (based on local CPU usage - not DB).

With solution i posted, request is 3-4x faster than original for local CPU.

2008/7/3 Andreas Pronakis <[hidden email]>:
I was wondering if anyone has tried using the module DBIx::Class::Cursor::Cached but replacing the cache engine Cache::FileCache with Memcache?

I will try it out and let you know the results, but if someone else has already tried it would be nice to know what their findings were

Andreas


--- On Wed, 6/25/08, Oleg Pronin <[hidden email]> wrote:

> From: Oleg Pronin <[hidden email]>
> Subject: Re: [Dbix-class] DBIx::Class and caching
> To: [hidden email], "DBIx::Class user and developer list" <[hidden email]>
> Date: Wednesday, June 25, 2008, 9:23 AM
> These approach caches ready objects, not just a data from
> database, so it
> quite fast, and lets you to cache any data with object,
> even not from DB.





_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Oleg Pronin
In reply to this post by Tobias Kremer
Of course when your project is not popular or you never run benchmark on what you write, you have no matter.
I have 5-7mln page hits per day. When i tried to use DBIC::Cursor::Cached, users were about to blow the server up.

2008/7/3 Tobias Kremer <[hidden email]>:
Quoting Andreas Pronakis <[hidden email]>:

> I was wondering if anyone has tried using the module
> DBIx::Class::Cursor::Cached but replacing the cache engine Cache::FileCache
> with Memcache?

Yes, we're using it in production for a site serving approx. 10 million
pageimpressions/month and it never failed on me. I've been told that it's also
in use at Takkle Inc. It works wonderfully for caching stuff that is supposed
to invalidate itself after a certain period of time. It's usage might be a bit
unfamiliar if you want to invalidate stuff manually because you have to have
the exact same ResultSet that was used to fill the cache in the first place.
Other than that I highly recommend its usage because at the moment it seems to
be the only useful working DBIC cache implementation.

I'm setting the cache_object attribute for all resultsets at once via the
schema's default_resultset_attributes() method.

--Tobias


_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Ash Berlin
In reply to this post by Oleg Pronin
On 3 Jul 2008, at 12:57, Oleg Pronin wrote:

DBIx::Class::Cursor::Cached is too slow for production use. It caches only data from DB making it impossible so store virtual data in objects and after fetching data from cache it has to construct a row object which is a expensive action for CPU, running all the 'inflate_result', and possibly other hard tasks.

When you use DBIC::C::C - it slows down the request by 2 times (based on local CPU usage - not DB).

With solution i posted, request is 3-4x faster than original for local CPU.

2008/7/3 Andreas Pronakis <[hidden email]>:
I was wondering if anyone has tried using the module DBIx::Class::Cursor::Cached but replacing the cache engine Cache::FileCache with Memcache?

I will try it out and let you know the results, but if someone else has already tried it would be nice to know what their findings were

Andreas


--- On Wed, 6/25/08, Oleg Pronin <[hidden email]> wrote:

> From: Oleg Pronin <[hidden email]>
> Subject: Re: [Dbix-class] DBIx::Class and caching
> To: [hidden email], "DBIx::Class user and developer list" <[hidden email]>
> Date: Wednesday, June 25, 2008, 9:23 AM
> These approach caches ready objects, not just a data from
> database, so it
> quite fast, and lets you to cache any data with object,
> even not from DB.


Please don't spread FUD.

Yes you can't store virtual data with Cursor::Cached, but then I'd argue that you shouldn't be doing that anyway.

And no it doesn't have to construct a row object if you don't want to See <http://search.cpan.org/~ash/DBIx-Class-0.08010/lib/DBIx/Class/ResultClass/HashRefInflator.pm> . Yes object construction can be expensive, but its what you want most of the time. If you dont want objects don't use an ORM. (The 'O' there stands for Object, dontcha know.)

Which solution? Link to archives please.

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Tobias Kremer
In reply to this post by Oleg Pronin
Quoting Oleg Pronin <[hidden email]>:
> Of course when your project is not popular or you never run benchmark on
> what you write, you have no matter.

This all depends on what you're caching. If your query runs for 0.001 seconds
there maybe is no need to cache it and doing so might indeed have a negative
impact on the overall performance. But if you generate lists that don't need to
be up-to-date with queries that run 1-2 seconds I'm pretty sure that
DBIC::Cursor::Cached WILL get you a performance boost no matter what! We
haven't seen a worrying drop in performance since using it.

> I have 5-7mln page hits per day. When i tried to use DBIC::Cursor::Cached,
> users were about to blow the server up.

Hits are not representative to boast with the performance of your Catalyst
application (http://en.wikipedia.org/wiki/Page_view). What percentage of those
hits are static files like images, css, js, etc which you shouldn't be serving
through catalyst anyway? 80%?

--Tobias

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Oleg Pronin
In reply to this post by Ash Berlin
I didn't want to affend you, sorry. But

>> but then I'd argue that you shouldn't be doing that anyway.
i dont agree. This is a great technic in many cases.

>> And no it doesn't have to construct a row object if you don't want to See
>> <http://search.cpan.org/~ash/DBIx-Class-0.08010/lib/DBIx/Class/ResultClass/HashRefInflator.pm>.

Then you will get a hash, not an object.

>> Yes object construction can be expensive, but its what you want most of the time. If you >> dont want objects don't use an ORM. (The 'O' there stands for Object, dontcha know.)

I posted a solution which caches already constructed objects in MEMD. You get objects! Without a need to construct then every time you get data from MEMD.
However DBIC::C::C will construct the same objects from DB data from cache THOUSAND TIMES, again and again - this is stupid! don't you see?
You think it is normal that when you start using C::C you get more local CPU usage than without it ? (it's not about Database CPU or disk usage, only perl-side).
I think it's NOT.

>> Which solution? Link to archives please.
somewhy i cannot find it in archives. I will repost it now.

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Oleg Pronin
In reply to this post by Tobias Kremer

DBIC::Cursor::Cached WILL get you a performance boost no matter what! We
haven't seen a worrying drop in performance since using it.

you didn't understand me. From the database side - you WILL get a perfomance boost.
but from LOCAL side (server where your backend is) you WILL LOSE perfomance (~ 2x slower).
Your perl program will take more CPU TIME (not wallclock time) than without Cursor::Cached at all.

I meant that my solution will get you a perfomance boost at both - perl-side and db-side

 

> I have 5-7mln page hits per day. When i tried to use DBIC::Cursor::Cached,
> users were about to blow the server up.

Hits are not representative to boast with the performance of your Catalyst
application (http://en.wikipedia.org/wiki/Page_view). What percentage of those
hits are static files like images, css, js, etc which you shouldn't be serving
through catalyst anyway? 80%?

--Tobias

I ment 5-7mln pure requests to catalyst backends (without any static).
 


_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Reply | Threaded
Open this post in threaded view
|

Re: DBIx::Class and caching

Matt S Trout
In reply to this post by Oleg Pronin
On Thu, Jul 03, 2008 at 03:57:11PM +0400, Oleg Pronin wrote:
> DBIx::Class::Cursor::Cached is too slow for production use. It caches only
> data from DB making it impossible so store virtual data in objects and after
> fetching data from cache it has to construct a row object which is a
> expensive action for CPU, running all the 'inflate_result', and possibly
> other hard tasks.

We've had this conversation.

Cursor::Cached is designed to deal with *DATABASE TIME LIMITED CODE*

Also, because it's transparent to the resultset, it can be used to cache
a result that will be used both as objects or via HashRefInflator.

It is -perfectly- usable in production -if- your constraints are appropriate.

You're right, if CPU time is also a time limit for the same code path then
DBIx::Cursor::Cached is inappropriate. There is a huge gap between "not
appropriate in all situations" and "not for production".

I would love to also see a DBIx::Class::ResultSet::Cached that would
save the full object data using $schema->freeze/thaw but I've seen first
hand that in some cases this is actually just as slow because of the time
spent to reconstruct the object from its cached form, and Cursor::Cached
was a simpler and sufficient solution for the use case it was written for.

But if you continue to make unqualified statements that things are unfit
for production when multiple sites are happily using them for the purpose
given and they are merely unuseful for -your- requirements, I am simply
going to remove you from this list permanently as a troll; I don't have
the energy to reply explaining the same fucking things every time you
make this sort of useless comment.

--
      Matt S Trout       Need help with your Catalyst or DBIx::Class project?
   Technical Director                    http://www.shadowcat.co.uk/catalyst/
 Shadowcat Systems Ltd.  Want a managed development or deployment platform?
http://chainsawblues.vox.com/            http://www.shadowcat.co.uk/servers/

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...