Topics

Filament•2y ago

Elastic Json Table

I'm trying solve a complicated problem and I'm looking for suggestions. I have an elastic index with 11.72 million docs I want to represent in a table. I created this index using fscrawler. It represents a single source of truth. However, filament tables are not really meant to handle this type. I've jumped a few hurdles using this post https://filamentphp.com/blog/how-to-consume-an-external-api-with-filament-tables. However, I need to filter that data by multiple searches and elastic can only query 10,000 records which you could increase, but it isn't 11 million. I think the path would be to create independent inputs per "column" and filter the data. So imagine a widget that feeds the table a query. Any questions or suggestions just looking for guidance.

42 Replies

Patrick Boivin•2y ago

So imagine a widget that feeds the table a query.

What would this look like? A form above the table so you can pre-fill some filters before the initial table query runs?

toeknee•2y ago

As above, I would recommend a filter form above ideally

Mack HOP•2y ago

@pboivin yes, that's how I had imagained it. Not sure how to do that, but seems logical.

Patrick Boivin•2y ago

I think it should be pretty straightforward, are you on a resource listing or a custom page?

Mack HOP•2y ago

That's a simple resource now.

Patrick Boivin•2y ago

Cool, let's see if I have something similar in my current project Sort of... I think the first step would be to create your Widget. Have you ever used the Forms package outside of the admin?

Mack HOP•2y ago

can't say that I have. It's all pretty new to me on the TALL side.

Patrick Boivin•2y ago

Ok, no worries. Start by creating your Widget, and add it to getHeaderWidgets() in your simple resource. The Widget itself is a Livewire component, so you can pick up the Forms docs here : https://filamentphp.com/docs/2.x/forms/getting-started#preparing-your-livewire-component You can define your custom form there and then, the hard part will be connecting it to the Page component, to interact with the table 😄

Mack HOP•2y ago

gives me a jumping off point thanks

Patrick Boivin•2y ago

Cool! Let me know when you're there, I'll help you with the table stuff (if I can...)

Mack HOP•2y ago

So we have an input box now. In order to load results we're calling a method on the model getRows(). How can I wire those two things up?

Mack HOP•2y ago

    public function getRows()
    {
        //API
        $originals = Http::get('http://127.0.0.1:9200/preservation/_search?q=*.*', [
            'size' => 10000,
            'query' => [
                'match_all' => [],
            ],
        ])->json();

        //filtering some attributes
        $originals = Arr::map($originals['hits']['hits'], function ($item) {
            //            dd($item['_id']);

            return [
                'id' => $item['_id'],
                'path' => str_replace($item['_source']['file']['filename'], '', $item['_source']['path']['virtual']),
                'filename' => $item['_source']['file']['filename'],
                'size' => $item['_source']['file']['filesize'],
                'created_at' => $item['_source']['file']['created'],
                'updated_at' => $item['_source']['file']['last_modified'],
                'type' => $item['_source']['file']['extension'],
            ];
        });

        return $originals;
    }

    public function getRows()
    {
        //API
        $originals = Http::get('http://127.0.0.1:9200/preservation/_search?q=*.*', [
            'size' => 10000,
            'query' => [
                'match_all' => [],
            ],
        ])->json();

        //filtering some attributes
        $originals = Arr::map($originals['hits']['hits'], function ($item) {
            //            dd($item['_id']);

            return [
                'id' => $item['_id'],
                'path' => str_replace($item['_source']['file']['filename'], '', $item['_source']['path']['virtual']),
                'filename' => $item['_source']['file']['filename'],
                'size' => $item['_source']['file']['filesize'],
                'created_at' => $item['_source']['file']['created'],
                'updated_at' => $item['_source']['file']['last_modified'],
                'type' => $item['_source']['file']['extension'],
            ];
        });

        return $originals;
    }

Patrick Boivin•2y ago

Oh, I forgot you are using Sushi... not sure if it'll make a difference Are you handling a submit event on the form? Something like this on the widget class

    public function submit()
    {
        $this->emitUp('updateGlobalFilter', $this->searchValue);
    }

    public function submit()
    {
        $this->emitUp('updateGlobalFilter', $this->searchValue);
    }

You can then catch this event in the Page component and dynamically update the table query

Mack HOP•2y ago

I prolly need to learn how to catch that event ha

Patrick Boivin•2y ago

https://laravel-livewire.com/docs/2.x/events#event-listeners

Livewire

Events | Livewire

A full-stack framework for Laravel that takes the pain out of building dynamic UIs.

Mack HOP•2y ago

This has been super helpful. I wasn't able to get emitUp to work, but emit does. Probably cause my listener is still on the component. Edit: moved it and emitUp works fine.

    protected function getTableQuery(): Builder
    {
        return parent::getTableQuery(); // TODO: Change the autogenerated stub
    }

    protected function getTableQuery(): Builder
    {
        return parent::getTableQuery(); // TODO: Change the autogenerated stub
    }

expects builder hmm..

Patrick Boivin•2y ago

Yeah so this might be a bit more complicated with Sushi, but let's see... Something like this should work with a regular model:

    protected function getTableQuery(): Builder
    {
        if (!$this->globalFilter) {
            // Force an empty query
            return parent::getTableQuery()->whereId('X');
        }

        return parent::getTableQuery()
            ->where('title', 'like', "%{$this->globalFilter}%");
    }

    protected function getTableQuery(): Builder
    {
        if (!$this->globalFilter) {
            // Force an empty query
            return parent::getTableQuery()->whereId('X');
        }

        return parent::getTableQuery()
            ->where('title', 'like', "%{$this->globalFilter}%");
    }

Mack HOP•2y ago

I like idea, but keep in mind we can only X amount of rows from the elastic API. I need to hit that getRows($query) function for a fresh call.

Patrick Boivin•2y ago

Yeah, that's the main issue, you want to pass the parameter to the Sushi, not modify the eloquent query

Mack HOP•2y ago

There is no database involved at all here.

Patrick Boivin•2y ago

I think I have something from a couple of months ago... 2 sec

Mack HOP•2y ago

    protected $listeners = ['searchOriginals' => 'refreshRows'];

    public function refreshRows($search = '')
    {
        $model = Originals::class;

        return $model::getRows($search);
    }

    protected $listeners = ['searchOriginals' => 'refreshRows'];

    public function refreshRows($search = '')
    {
        $model = Originals::class;

        return $model::getRows($search);
    }

getRows returns correct data, but doesn't refresh table.

Patrick Boivin•2y ago

I don't have a ton of experience with Sushi, so there may be a better way. But this how I'm passing parameters to my Sushi model currently:

class ProductCompatibleVehicle extends Model
{
    use Sushi;

    protected static array $vehicle_ids = [];

    public static function queryForVehicleIds(array $ids): Builder
    {
        static::$vehicle_ids = $ids;

        return static::query();
    }

    public function getRows()
    {
        if (!self::$vehicle_ids) {
            return [];
        }

        // ... prepare rows ...
        
        return $rows;
    }
}

class ProductCompatibleVehicle extends Model
{
    use Sushi;

    protected static array $vehicle_ids = [];

    public static function queryForVehicleIds(array $ids): Builder
    {
        static::$vehicle_ids = $ids;

        return static::query();
    }

    public function getRows()
    {
        if (!self::$vehicle_ids) {
            return [];
        }

        // ... prepare rows ...
        
        return $rows;
    }
}

Maybe something like this for your use case:

    protected function getTableQuery(): Builder
    {
        return Originals::queryWithSearch($this->searchValue);
    }

    protected function getTableQuery(): Builder
    {
        return Originals::queryWithSearch($this->searchValue);
    }

In your event listener, save the search value to a public property instead ^ Then the table will refresh automatically if you setup getTableQuery()

Mack HOP•2y ago

Okay, that should be helpful. How does getRows get called here though? Asking from ignorance

Patrick Boivin•2y ago

It gets called behind the scenes, you don't usually have to call it yourself. The sushi model acts like a regular model, so the filament table will build the query from what it finds in getTableQuery()

Mack HOP•2y ago

very close.

    protected static string $original_search = '';

    public static function queryForOriginalSearch($search): Builder
    {
        static::$original_search = $search;

        return static::query();
    }

    public function getRows()
    {
        //self::$original_search = 'news'; this works

        if (! self::$original_search) {
            return [];
        }

        //stuff

        return $originals;
    }

    protected static string $original_search = '';

    public static function queryForOriginalSearch($search): Builder
    {
        static::$original_search = $search;

        return static::query();
    }

    public function getRows()
    {
        //self::$original_search = 'news'; this works

        if (! self::$original_search) {
            return [];
        }

        //stuff

        return $originals;
    }

if I dump static::$original_search it has the value, but seems to fail right at the static::query() part. wehen I dump static::query() it's just an instance for Builder for the originals model.

Illuminate\Database\Eloquent\Builder {#2145 ▼ // app/Models/Originals.php:27
  #query: Illuminate\Database\Query\Builder {#1884 ▶
    +connection: Illuminate\Database\SQLiteConnection {#2130 …24}
    +grammar: Illuminate\Database\Query\Grammars\SQLiteGrammar {#1894 …5}
    +processor: Illuminate\Database\Query\Processors\SQLiteProcessor {#1992}
    +bindings: array:9 [▶
      "select" => []
      "from" => []
      "join" => []
      "where" => []
      "groupBy" => []
      "having" => []
      "order" => []
      "union" => []
      "unionOrder" => []
    ]
....

Illuminate\Database\Eloquent\Builder {#2145 ▼ // app/Models/Originals.php:27
  #query: Illuminate\Database\Query\Builder {#1884 ▶
    +connection: Illuminate\Database\SQLiteConnection {#2130 …24}
    +grammar: Illuminate\Database\Query\Grammars\SQLiteGrammar {#1894 …5}
    +processor: Illuminate\Database\Query\Processors\SQLiteProcessor {#1992}
    +bindings: array:9 [▶
      "select" => []
      "from" => []
      "join" => []
      "where" => []
      "groupBy" => []
      "having" => []
      "order" => []
      "union" => []
      "unionOrder" => []
    ]
....

Patrick Boivin•2y ago

Yes that's normal So the table will get the Builder instance and prepare the final query. Then behind the scenes, Sushi will hit the getRows() method which is what it uses instead of a DB table.

Mack HOP•2y ago

It seems to never do that last step

Patrick Boivin•2y ago

What do you have in getTableQuery() ?

Mack HOP•2y ago

    public function getTableQuery(): Builder
    {
        return Originals::queryForOriginalSearch($this->search);
    }

    public function getTableQuery(): Builder
    {
        return Originals::queryForOriginalSearch($this->search);
    }

the whole widget

    use Forms\Concerns\InteractsWithForms;

    protected int|string|array $columnSpan = 'full';

    public string $search = '';

    public $listeners = ['searchOriginals' => 'getTableQuery'];

    public function getTableQuery(): Builder
    {
        return Originals::queryForOriginalSearch($this->search);
    }

    protected function getFormSchema(): array
    {
        return [
            Forms\Components\TextInput::make('search'),
        ];
    }

    public function submit()
    {
        $this->emit('searchOriginals', $this->search);
    }

    public function render(): View
    {
        return view('filament.widgets.original-search');
    }

    use Forms\Concerns\InteractsWithForms;

    protected int|string|array $columnSpan = 'full';

    public string $search = '';

    public $listeners = ['searchOriginals' => 'getTableQuery'];

    public function getTableQuery(): Builder
    {
        return Originals::queryForOriginalSearch($this->search);
    }

    protected function getFormSchema(): array
    {
        return [
            Forms\Components\TextInput::make('search'),
        ];
    }

    public function submit()
    {
        $this->emit('searchOriginals', $this->search);
    }

    public function render(): View
    {
        return view('filament.widgets.original-search');
    }

Patrick Boivin•2y ago

Ah ok, do the listener is the issue I think 1 sec Whoa, wait. This is all in the widget? Do you have anything on the Page component? This is what I have in mind for the widget:

    public string $search = '';

    protected function getFormSchema(): array
    {
        return [
            Forms\Components\TextInput::make('search'),
        ];
    }

    public function submit()
    {
        $this->emit('searchOriginals', $this->search);
    }

    public function render(): View
    {
        return view('filament.widgets.original-search');
    }

    public string $search = '';

    protected function getFormSchema(): array
    {
        return [
            Forms\Components\TextInput::make('search'),
        ];
    }

    public function submit()
    {
        $this->emit('searchOriginals', $this->search);
    }

    public function render(): View
    {
        return view('filament.widgets.original-search');
    }

Mack HOP•2y ago

Okay, I can move the rest to ManageOriginals

Patrick Boivin•2y ago

I think for Manage... :

class ManageOriginals extends ManageRecords
{
    public string $search = '';

    protected function getListeners()
    {
        return ['searchOriginals' => 'updateSearchValue'];
    }

    public function updateSearchValue($value): void
    {
        $this->search = $value;
    }

    protected function getTableQuery(): Builder
    {
        return Originals::queryForOriginalSearch($this->search);
    }
}

class ManageOriginals extends ManageRecords
{
    public string $search = '';

    protected function getListeners()
    {
        return ['searchOriginals' => 'updateSearchValue'];
    }

    public function updateSearchValue($value): void
    {
        $this->search = $value;
    }

    protected function getTableQuery(): Builder
    {
        return Originals::queryForOriginalSearch($this->search);
    }
}

Mack HOP•2y ago

Today has been a lesson.

Patrick Boivin•2y ago

For sure! Covered a lot of ground with this feature 😄

Mack HOP•2y ago

Yeah you're kind of awesome, but if anybody asks I'll deny it.

Patrick Boivin•2y ago

I'll take the compliment for now 👌

Mack HOP•2y ago

Mack HOP•2y ago

@pboivin what do you think the best approach to having multiple search boxes? aka table filtering ha

Patrick Boivin•2y ago

You can bind the entire form to a single property (array), so if you have multiple fieds, you can still emit just the single array to the page component and pass it along to the Sushi model https://filamentphp.com/docs/2.x/forms/getting-started#scoping-form-data-to-an-array-property

awcodes•2y ago

Been following this today. Really impressive.

Mack HOP•2y ago

@pboivin legit I have learned more today than I have in a crazy long time. Thank you so much for taking the time out. I owe you a cold 🍺 or 10.

Filament is a collection of beautiful full-stack components for Laravel.You can get help using it on our Discord.

18KMembers

View on Discord

Want results from more Discord servers?

Add your server